From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 533FC1A109; Mon, 08 Sep 2025 10:00:05 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 993) id C86DB19FEF; Mon, 08 Sep 2025 10:00:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_MISSING,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=4.0.1 Received: from fout-b5-smtp.messagingengine.com (fout-b5-smtp.messagingengine.com [202.12.124.148]) by atuin.qyliss.net (Postfix) with ESMTPS id 9CDBD19FEB for ; Mon, 08 Sep 2025 10:00:00 +0000 (UTC) Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfout.stl.internal (Postfix) with ESMTP id 7E6121D00120; Mon, 8 Sep 2025 05:59:59 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Mon, 08 Sep 2025 05:59:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1757325599; x=1757411999; bh=zfs6l8DnUq uifaHXlEBdaFXfuwK1kM0xg7ymrY+cmD0=; b=l7gzE8TKR20l4oeOjYajsWCx1q YHbm0ZLgngnRuzUC9N0JtSNc8Fz9jGixuEUbBCTWiwCoeL1BN1lij8bgImq5lPyL N2qyTtxCu3QaH+0CtGBLWaLToHzIzfogiELP33aTwygVErapS+89PeUKi+Pj9N4T wvtsqd4r/yunVqhbLXTkbQ7k8PA44ky2H7E3xCLh5peNXXedc4dvzBCXr3aOY6bv WtSEFdFsG2D7Q4VT4D7vr3mkwwzaZ7XNx/kX1AE7CYGD6kkZ5Bsk9HP5opt1mhuY 8MAJr4mbhXv770V1kDtmpTvOWswpOJv49r+4dCq8+WPvQSfD0HDOnoiriFAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1757325599; x=1757411999; bh=zfs6l8DnUquifaHXlEBdaFXfuwK1kM0xg7y mrY+cmD0=; b=n95hKSlQQUL9qh4KAHhNdV51LPozB0Tmw8z4qK1LT6eotGCkuBi Mejew7roOM3MGnzevvzw2+I9vazQoU7xT15W3CopJJw098nvnqOo5eSR0m+tMoOO BVEgAMT+javHh9vbxdrHEFTnqtz7VDofaDeQkd3qYXNjyoZePYeiytVYtF+H5FFs +t7VNAi8t2oD1CM+pdDIuWWxEZ1pSx8GBmHqk0BMmvj+KdOXoDu5tpQms7OvqTo0 QB2lq9W80VDyB2T2f6qM7bAdVsTq7WGadF23TAkm8oU1GRk0eDdf4ciRbTevCl8o XqrfxycIcsZpgAYW0nYRtH9efJyl3UsdWJA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggddujedvfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufgjfhffkfggtgesghdtreertddttdenucfhrhhomheptehlhihsshgrucft ohhsshcuoehhihesrghlhihsshgrrdhisheqnecuggftrfgrthhtvghrnhepieduffeuie elgfetgfdttddtkeekheekgfehkedufeevteegfeeiffetvdetueevnecuvehluhhsthgv rhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhephhhisegrlhihshhsrgdrih hspdhnsggprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegu vghmihhosggvnhhouhhrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepuggvvhgvlhessh hpvggtthhruhhmqdhoshdrohhrgh X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 8 Sep 2025 05:59:58 -0400 (EDT) Received: by mbp.qyliss.net (Postfix, from userid 1000) id 7210C1675ADF; Mon, 08 Sep 2025 11:59:55 +0200 (CEST) From: Alyssa Ross To: Demi Marie Obenour Subject: Re: [PATCH 2/4] Generate makefile file lists from a script In-Reply-To: <20250903-genfiles-v1-2-cc993fcb1e4c@gmail.com> References: <20250903-genfiles-v1-0-cc993fcb1e4c@gmail.com> <20250903-genfiles-v1-2-cc993fcb1e4c@gmail.com> Date: Mon, 08 Sep 2025 11:59:54 +0200 Message-ID: <87zfb5i7s5.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Message-ID-Hash: FIKEWICV2JWKLA6C2A5KB25MMO6EW3K7 X-Message-ID-Hash: FIKEWICV2JWKLA6C2A5KB25MMO6EW3K7 X-MailFrom: hi@alyssa.is X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; header-match-devel.spectrum-os.org-4; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Spectrum OS Development X-Mailman-Version: 3.3.9 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Demi Marie Obenour writes: > The script will always get them right, whereas humans (the author of > this commit included) generally will not. > > Signed-off-by: Demi Marie Obenour I like the idea! > --- > Documentation/development/built-in-vms.adoc | 7 ++ > host/rootfs/Makefile | 107 ++--------------------= ---- > host/rootfs/file-list.mk | 100 ++++++++++++++++++++++= ++ > img/app/Makefile | 74 +++--------------- > img/app/file-list.mk | 65 ++++++++++++++++ > lib/common.mk | 1 + > scripts/genfiles.awk | 115 ++++++++++++++++++++++= ++++++ > vm/sys/net/Makefile | 51 +++--------- > vm/sys/net/file-list.mk | 41 ++++++++++ > 9 files changed, 357 insertions(+), 204 deletions(-) > > diff --git a/Documentation/development/built-in-vms.adoc b/Documentation/= development/built-in-vms.adoc > index e90009ee5a3c2c254a7ae11e36121576b819eee7..82d78705a6020bbdb06fbc123= a32dbdd6fd50085 100644 > --- a/Documentation/development/built-in-vms.adoc > +++ b/Documentation/development/built-in-vms.adoc > @@ -44,6 +44,13 @@ NOTE: As a special convenience, it's not necessary to = run `make clean` > if the only change to the Nix files is modifying the packages > installed in the VM. >=20=20 > +The list of files used for the VM image is stored in a separate file, > +`file-lists.mk`. You can edit it manually when developing. However, > +after each commit that adds or removes files from it, you should run > +`make update-file-list`, which will regenerate it from the output of > +`git ls-files`. Any changes you made will be lost. This ensures > +that the file lists are always in sync with the git repository. > + TBH editing it manually and then losing your changes is probably going to be more of a footgun than anything else. You can get the same result by staging new files and rerunning the script, so I'd avoid mentioning the manual editing option, especially given you have a "DO NOT EDIT" comment. > -$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(addprefix image/= ,$(FILES)) $(BUILD_FILES) build/empty build/fifo > +$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(addprefix image/= ,$(FILES)) $(BUILD_FILES) build/empty build/fifo file-list.mk Given that we don't include Makefile as a dependency, it probably doesn't make sense to depend on other included Makefile fragments either? > @@ -207,6 +111,11 @@ debug: > $(VMLINUX) > .PHONY: debug >=20=20 > +update-file-list: > + ../../scripts/genfiles.awk image > file-list.mk > + > +.PHONY: update-file-list > + Given this doesn't use any features of Make, it probably makes more sense to just run the script directly. It could output into file-list.mk by default for ergonomics. > run: build/live.img $(EXT_FS) build/rootfs.verity.roothash > @set -x && \ > ext=3D"$$(mktemp build/spectrum-rootfs-extfs.XXXXXXXXXX.img)" && \ > diff --git a/host/rootfs/file-list.mk b/host/rootfs/file-list.mk > new file mode 100644 > index 0000000000000000000000000000000000000000..0817887d0bb25ab47e777f6a1= 30a3b6214b25f0f > --- /dev/null > +++ b/host/rootfs/file-list.mk > @@ -0,0 +1,100 @@ > +# SPDX-License-Identifier: CC0-1.0 > +# SPDX-FileCopyRightText: Not Copyrightable (machine-written) SPDX-FileCopy*r*ightText, and should probably say that you're the owner, for consistency with e.g. lib/nixpkgs.default.nix, which is also generated. You at least made the template. > +# Generated by scripts/genfile.awk, DO NOT EDIT! > +override FILES ::=3D \ Our Makefiles are POSIX. (Mostly because it's the only sensible way to draw the line, and keep all the really advanced easy to misuse GNU stuff out.) > diff --git a/lib/common.mk b/lib/common.mk > index 277c3544036d9a9057f8ba4ad37fe2207548cc59..0a03ff440cc671264d2b859a2= ae048db9252d047 100644 > --- a/lib/common.mk > +++ b/lib/common.mk > @@ -1,5 +1,6 @@ > # SPDX-License-Identifier: EUPL-1.2+ > # SPDX-FileCopyrightText: 2021, 2023, 2025 Alyssa Ross > +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour >=20=20 > BACKGROUND =3D background > CPIO =3D cpio This change looks like an accident? > diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk > new file mode 100755 > index 0000000000000000000000000000000000000000..62863e78f157f1d9a0f6dbdb0= f4380db9c9d48cb > --- /dev/null > +++ b/scripts/genfiles.awk > @@ -0,0 +1,115 @@ > +#!/usr/bin/env -S LC_ALL=3DC LANGUAGE=3DC awk -E > +# SPDX-License-Identifier: EUPL-1.2+ > +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour > +function check_status(status) { > + if (status < 0) { > + printf "FATAL: getline: %s\n", status > "/dev/stderr"; > + exit 1; > + } > + return status; > +} > + > +function check_close(value, status) { > + status =3D check_status(close(value)); > + if (status !=3D 0) { > + printf "FATAL: command exited with status %d\n", status > "/dev/stderr= "; > + exit status; > + } > +} > + > +function shell_quote(command) { > + gsub(/'/, "'\\\\&'", command); > + return ("'" command "'"); > +} > + > +function get(command, line, path, array_index, inode_type, mode= , modes, symlink_count, symlinks, file_count, files, rc_count, rc_files, is= _license, is_rc) { > + file_count =3D 0; > + symlink_count =3D 0; > + rc_count =3D 0; > + modes["120000"] =3D "symlink"; > + modes["040644"] =3D "directory"; > + modes["040755"] =3D "directory"; > + modes["100644"] =3D "regular"; > + modes["100755"] =3D "regular"; > + print "# SPDX-License-Identifier: CC0-1.0"; > + print "# SPDX-FileCopyRightText: Not Copyrightable (machine-written)"; > + print "# Generated by scripts/genfile.awk, DO NOT EDIT!"; > + while (check_status(command | getline line)) { > + if (line !~ /^[0-7]{6}\t/) { > + # this is a git bug > + print "FATAL: git ls-files output didn't start with a valid mode" > "= /dev/stderr"; > + exit 1; > + } > + path =3D substr(line, 8); > + if (path !~ /^[ -~]+$/) { > + # also a git bug > + print "FATAL: git ls-files didn't quote properly" > "/dev/stderr"; > + exit 1; > + } > + if (path ~ /^\/|((^|\/)\.{0,2}($|\/))/) { > + # also a git bug > + printf "FATAL: git ls-files output non-canonical path '%s'\n", path >= "/dev/stderr"; > + exit 1; > + } > + if (path !~ /^[[:alnum:]_.+@/-]+$/) { > + printf "FATAL: filename '%s' has forbidden characters\n", path > "/de= v/stderr"; > + exit 1; > + } I feel like this could be a lot nicer if we ran git ls-files outside awk, and could then use its nice top-level matching syntax? > + mode =3D modes[substr(line, 1, 6)]; > + is_license =3D path ~ /\.license$/; > + is_rc =3D path ~ /^etc\/s6-rc\//; > + if (mode =3D=3D "regular") { > + if (is_license) { > + continue; > + } > + if (is_rc) { > + rc_count +=3D 1; > + rc_files[rc_count] =3D path; > + } else { > + file_count +=3D 1; > + files[file_count] =3D path; > + } > + continue; > + } > + if (mode =3D=3D "symlink") { > + if (is_rc) { > + printf "FATAL: symlink in s6-rc-compile input: %s\n", path; > + exit 1; > + } > + symlink_count +=3D 1; > + symlinks[symlink_count] =3D path; > + } else if (mode !=3D "directory") { > + printf "FATAL: file %s has unknown mode %s\n", path, substr(line, 1, = 6) > "/dev/stderr"; > + exit 1; > + } > + if (is_license) { > + printf "FATAL: %s (type %s) ends in .license\n", path, mode > "/dev/s= tderr"; > + exit 1; > + } > + } > + check_close(command); > + > + printf "override FILES ::=3D"; > + for (array_index =3D 1; array_index <=3D file_count; array_index +=3D 1= ) { > + printf " \\\n\t%s", files[array_index]; > + } > + printf ("\n\n" \ > +"# These are separate because they need to be included, but putting\n" \ > +"# them as make dependencies would confuse make.\n" \ > +"override LINKS ::=3D"); > + for (array_index =3D 1; array_index <=3D symlink_count; array_index += =3D 1) { > + printf " \\\n\t%s", symlinks[array_index]; > + } > + printf "\n\noverride S6_RC_FILES ::=3D"; > + for (array_index =3D 1; array_index <=3D rc_count; array_index +=3D 1) { > + printf " \\\n\t%s", rc_files[array_index]; > + } > + printf "\n" > +} > + > +BEGIN { > + RS =3D "\n"; > + FS =3D "\t"; > + get("set -euo pipefail && { git -c core.quotePath=3Dtrue -C " shell_quo= te(ARGV[1]) " ls-files '--format=3D%(objectmode)\t%(path)' -- .|sort -t '\t= ' -k 2;}"); > + exit 0; > +} --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRV/neXydHjZma5XLJbRZGEIw/wogUCaL6pGgAKCRBbRZGEIw/w onhbAQC5rwe8uboQns41RA2dCf8jIlUn6Euiv/xB8i580rD4jQEAtE3LePi2L9iq HS2ySwdhJqgDvEp8h4QU0XE9pFVuGgs= =KklI -----END PGP SIGNATURE----- --=-=-=--