[PATCH] scripts/make-gpt.sh: reserve 8 MiB at start of GPT

[PATCH] scripts/make-gpt.sh: reserve 8 MiB at start of GPT

[Alyssa Ross]

In some cases (e.g. a product built on Spectrum where updates to both
firmware and OS are delivered by the same vendor), it makes sense to
include firmware in the same image as the OS.  This firmware tends to
live in the start of the image, after the GPT header.  (The exact
location varies by board.)

The 1 MiB reserved by sfdisk by default is not enough for this.  The
conventional value seems to be 8 MiB.  On NixOS, this is configurable
with the sdImage.firmwarePartitionOffset option, but this defaults to
8 MiB and I wasn't able to find any examples in searches on DuckDuckGo
and GitHub of anybody using any other value, so it looks like 8 MiB is
broadly acceptable.

Signed-off-by: Alyssa Ross <hi@alyssa.is>
---
Yuriy, if you'd like to test this patch, here's some documentation
about how to do that. :)

https://spectrum-os.org/doc/testing-patches.html

 scripts/make-gpt.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/make-gpt.sh b/scripts/make-gpt.sh
index 56076d3..55f38d6 100755
--- a/scripts/make-gpt.sh
+++ b/scripts/make-gpt.sh
@@ -1,12 +1,13 @@
 #!/bin/sh -eu
 #
 # SPDX-FileCopyrightText: 2021-2022 Alyssa Ross <hi@alyssa.is>
+# SPDX-FileCopyrightText: 2022 Unikie
 # SPDX-License-Identifier: EUPL-1.2+
 #
 # usage: make-gpt.sh GPT_PATH PATH:PARTTYPE[:PARTUUID]...
 
 ONE_MiB=1048576
-TWO_MiB=2097152
+NINE_MiB=9437184
 
 # Prints the number of 1MiB blocks required to store the file named
 # $1.  We use 1MiB blocks because that's what sfdisk uses for
@@ -41,8 +42,9 @@ shift
 nl=$'\n'
 table="label: gpt"
 
-# Keep 1MiB free at the start, and 1MiB free at the end.
-gptBytes=$TWO_MiB
+# Keep 8MiB free at the start (to provide space for firmware if
+# required), and 1MiB free at the end.
+gptBytes=$NINE_MiB
 for partition; do
 	sizeMiB="$(sizeMiB "$(partitionPath "$partition")")"
 	table="$table${nl}size=${sizeMiB}MiB,$(awk -f "$scriptsDir/sfdisk-field.awk" -v partition="$partition")"
@@ -52,6 +54,7 @@ done
 rm -f "$out"
 truncate -s "$gptBytes" "$out"
 sfdisk "$out" <<EOF
+first-lba: 16384
 $table
 EOF
 
-- 
2.37.1

Re: [PATCH] scripts/make-gpt.sh: reserve 8 MiB at start of GPT

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]

On Fri, Aug 26, 2022 at 01:09:25PM +0000, Alyssa Ross wrote:
> In some cases (e.g. a product built on Spectrum where updates to both
> firmware and OS are delivered by the same vendor), it makes sense to
> include firmware in the same image as the OS.  This firmware tends to
> live in the start of the image, after the GPT header.  (The exact
> location varies by board.)
>
> The 1 MiB reserved by sfdisk by default is not enough for this.  The
> conventional value seems to be 8 MiB.  On NixOS, this is configurable
> with the sdImage.firmwarePartitionOffset option, but this defaults to
> 8 MiB and I wasn't able to find any examples in searches on DuckDuckGo
> and GitHub of anybody using any other value, so it looks like 8 MiB is
> broadly acceptable.

Update: following some discussion in #spectrum, I'm now aware of a
couple of changes we'd want to make if we went down this path:

 * We should use a "protective partition" in the GPT to identify
   firmware regions, rather than just offsetting the first partition,
   so it's clearer that that space is there and what it's used for, as
   described in the Embedded Boot Base Requirements (EBBR) spec[1].

 * 8 MiB isn't enough for all boards.  Specifically, with Rockchip +
   mainline U-Boot, the U-Boot image _starts_ at 8 MiB, so we'd want to
   reserve at least 16 MiB, and possibly more for forward compatibility.

But another approach that was identified, that might work better, would
be for "ports" of Spectrum that want to install firmware in the same
image to just make their own GPTs, with space reserved for firmware as
required, and copy the partitions from the Spectrum image into this new
partition table.  This way, we can avoid trying to find a
one-size-fits-all solution to a problem that varies by platform, and
avoid introducing configuration knobs unnecessarily.  In Spectrum,
partitions are always identified by UUID anyway, so it wouldn't matter
if a downstream or end user moves them around as required.  And we'll
need to end up doing something like this at some point _anyway_, because
to support dual boot setups our installer would need to be able to copy
Spectrum partitions into place on the user's existing GPT.

So I think that should be the path forward for Spectrum product images:
have a script that creates a GPT, adds a partition for the platform
firmware, then copies all the partitions from the Spectrum images into
place after it.

But as always, I'm open to further discussion if there's a problem with
this approach. :)

[1]: https://arm-software.github.io/ebbr/index.html#partitioning-of-shared-storage

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scripts/make-gpt.sh: reserve 8 MiB at start of GPT

[Ville Ilvonen]

On 8/26/22 16:09, Alyssa Ross wrote:
> In some cases (e.g. a product built on Spectrum where updates to both
> firmware and OS are delivered by the same vendor), it makes sense to
> include firmware in the same image as the OS.  This firmware tends to
> live in the start of the image, after the GPT header.  (The exact
> location varies by board.)
> 
> The 1 MiB reserved by sfdisk by default is not enough for this.  The
> conventional value seems to be 8 MiB.  On NixOS, this is configurable
> with the sdImage.firmwarePartitionOffset option, but this defaults to
> 8 MiB and I wasn't able to find any examples in searches on DuckDuckGo
> and GitHub of anybody using any other value, so it looks like 8 MiB is
> broadly acceptable.
> 
> Signed-off-by: Alyssa Ross <hi@alyssa.is>
> ---

Thanks, this is an enabler for cross-compilation build configs for imx8 
boards. Works as intended to enable the image building using
https://github.com/tiiuae/spectrum-config-imx8 for imx8qxp but
requires more patch(es) to Spectrum img/live/Makefile (pending).

Testing

Without:
1527808 bytes (1.5 MB, 1.5 MiB) copied, 0.00650894 s, 235 MB/s
init: sector size too big
Cannot initialize '::'
Bad target ::/

With:
1527808 bytes (1.5 MB, 1.5 MiB) copied, 0.00981576 s, 156 MB/s
/nix/store/609b8cnks906dkpcq8hn6v0b1farbis2-spectrum-live-imx8qxp.img-0.1

Tested-by: Ville Ilvonen <ville.ilvonen@unikie.com>

-Ville


> Yuriy, if you'd like to test this patch, here's some documentation
> about how to do that. :)
> 
> https://spectrum-os.org/doc/testing-patches.html
> 
>   scripts/make-gpt.sh | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/make-gpt.sh b/scripts/make-gpt.sh
> index 56076d3..55f38d6 100755
> --- a/scripts/make-gpt.sh
> +++ b/scripts/make-gpt.sh
> @@ -1,12 +1,13 @@
>   #!/bin/sh -eu
>   #
>   # SPDX-FileCopyrightText: 2021-2022 Alyssa Ross <hi@alyssa.is>
> +# SPDX-FileCopyrightText: 2022 Unikie
>   # SPDX-License-Identifier: EUPL-1.2+
>   #
>   # usage: make-gpt.sh GPT_PATH PATH:PARTTYPE[:PARTUUID]...
>   
>   ONE_MiB=1048576
> -TWO_MiB=2097152
> +NINE_MiB=9437184
>   
>   # Prints the number of 1MiB blocks required to store the file named
>   # $1.  We use 1MiB blocks because that's what sfdisk uses for
> @@ -41,8 +42,9 @@ shift
>   nl=$'\n'
>   table="label: gpt"
>   
> -# Keep 1MiB free at the start, and 1MiB free at the end.
> -gptBytes=$TWO_MiB
> +# Keep 8MiB free at the start (to provide space for firmware if
> +# required), and 1MiB free at the end.
> +gptBytes=$NINE_MiB
>   for partition; do
>   	sizeMiB="$(sizeMiB "$(partitionPath "$partition")")"
>   	table="$table${nl}size=${sizeMiB}MiB,$(awk -f "$scriptsDir/sfdisk-field.awk" -v partition="$partition")"
> @@ -52,6 +54,7 @@ done
>   rm -f "$out"
>   truncate -s "$gptBytes" "$out"
>   sfdisk "$out" <<EOF
> +first-lba: 16384
>   $table
>   EOF
>