From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id BF7D423BC4; Sat, 27 Sep 2025 08:19:55 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 993) id 8038923BB0; Sat, 27 Sep 2025 08:19:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_MISSING,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=4.0.1 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) by atuin.qyliss.net (Postfix) with ESMTPS id 804E523BAE for ; Sat, 27 Sep 2025 08:19:51 +0000 (UTC) Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 6D1D614000A4; Sat, 27 Sep 2025 04:19:50 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Sat, 27 Sep 2025 04:19:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1758961190; x=1759047590; bh=Bjt4Vea0ZR BRLRrdWLAwD5LYQT6bSP6TXf7GNgqs6D0=; b=JpXCvNnHYtCSA5vCQRMFPuQTQD hjjk0teJw6VMlhW/X5xS9zJzfCjpfEoOKYmKKCMfwwYVRNgYLx/1PSX5mXF4vALH YTCvF1fdddkKu7zk+Dwpg+56Pd9xVHIsFbYs+cb9Fbfcq8wY1gGohmpnTY2OMryP Qx8Q/hZHXIPrUSG1Ma2yyqwN12uC6iQ2SQTcTvSAiOyiQ4lxdnE7bOY8Q/uFsXJt aIh857ZKayCzOZyV+z9V+LixyfZ2V8JUOTKfIu+7t80/YCYSTXAOkPiugK9smGUk bpSMAXqHYs4aGYiLHR4gB/lQEpRJQdj+JtA4fCke0XsdzbomMjF+nvs9/GeA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1758961190; x=1759047590; bh=Bjt4Vea0ZRBRLRrdWLAwD5LYQT6bSP6TXf7 GNgqs6D0=; b=ByWrQy3D3afkMtPAWZ0nx+VGSarHkCWWrGfLoy6jDXEOepJSfM7 dYBgakL4o87KAlKTEBDoRUCqaQawRYsZJ9jdJumbqFXEybYZUlARkAMYmH718Qsr 0cUNltOPnY81nhZA8L1unlpFnGwP5aT+sVGBQhToDplY1Qpw0GtsKuCWW98Ji/J4 /VcuchxZWOwqMV/B2NCB1sXzCfdOvs0RtghpHQSlFkz1yhY9+p5Baq4lXY3qNW5R 5wOU4K27Lvx2QV2gP65kYcQ5PKk9u/iueSCmUkYr18KJK2iV+f81aY4uYeK8PVlC kmEEOgfDien0p4DvZMWAsNN0R5avD3KqWVA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdejudejhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufgjfhffkfggtgesghdtreertddttdenucfhrhhomheptehlhihsshgrucft ohhsshcuoehhihesrghlhihsshgrrdhisheqnecuggftrfgrthhtvghrnhepuedtueetle ejgeejhfefjeelfeduleevledtiefhgfdutdelveevheeiueevfeejnecuffhomhgrihhn pehkvghrnhgvlhdrohhrghdpshhpvggtthhruhhmqdhoshdrohhrghenucevlhhushhtvg hrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehhihesrghlhihsshgrrdhi shdpnhgspghrtghpthhtohepvddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepug gvmhhiohgsvghnohhurhesghhmrghilhdrtghomhdprhgtphhtthhopeguvghvvghlsehs phgvtghtrhhumhdqohhsrdhorhhg X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 27 Sep 2025 04:19:49 -0400 (EDT) Received: by mbp.qyliss.net (Postfix, from userid 1000) id 8D35224D8526; Sat, 27 Sep 2025 10:19:47 +0200 (CEST) From: Alyssa Ross To: Demi Marie Obenour Subject: Re: [PATCH v5] Generate file lists from a script In-Reply-To: <20250926-genfiles-v5-1-3394dfb3b330@gmail.com> References: <20250921-genfiles-v4-1-4375bda78707@gmail.com> <20250926-genfiles-v5-1-3394dfb3b330@gmail.com> Date: Sat, 27 Sep 2025 10:19:46 +0200 Message-ID: <87tt0ob93x.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Message-ID-Hash: DFHAFVMHU6MYG63NLFNZP4AA2NQIX5MU X-Message-ID-Hash: DFHAFVMHU6MYG63NLFNZP4AA2NQIX5MU X-MailFrom: hi@alyssa.is X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; header-match-devel.spectrum-os.org-4; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Spectrum OS Development X-Mailman-Version: 3.3.9 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --=-=-= Content-Type: text/plain Demi Marie Obenour writes: > Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have > manually-maintained lists of files and symlinks. These duplicate the > information in the git repository and can easily get out of sync or > cause unnecessary merge conflicts. Fix all of these issues by having > the git repository be the source of truth, and using a script to > generate the file lists. Developers can regenerate the lists before > every commit, or even add a git hook to do that. > > Signed-off-by: Demi Marie Obenour > --- > This actually reduces the amount of code that has to be written by hand. > --- This is so close. Was almost at the point where I could have just fixed it up myself and committed it, but there's one thing I want to check with you. > Changes in v5: > - Use 'print ""' instead of 'print' in awk to print a newline. 'print' > with no arguments implicitly prints $0 instead. This caused the > generated makefiles to be incorrect. > - Use S6_RC_FILES instead of VM_S6_RC_FILES in vm/sys/net/Makefile and > img/app/Makefile. This prevented the image from being built. > - Do not check for git repository being in a directory with a name > ending in a newline. > - Use shell redirection instead of awk redirection. > - Do not include trailing DONE line in input to awk. > - Link to v4: https://lore.kernel.org/r/20250921-genfiles-v4-1-4375bda78707@gmail.com > > Changes in v4: > - Use /bin/sh instead of bash. > - Do not assume that negated awk character classes match all bytes. > - Do not check the mode of license files. > - Use implicit awk variable initialization. > - Use 'git rev-parse --show-toplevel' to find the repository root. > - Remove wrongly added copyright header. > - Improve documentation. > - Remove git hooks. > - Add missing copyright header. > - Avoid non-portable /usr/bin/env -S. > - Avoid assuming that awk is GNU awk. > - Avoid non-portable awk -E. > - Do not check for git bugs. > - Fix link in v3 changelog. > - Link to v3: https://spectrum-os.org/lists/archives/spectrum-devel/20250920-genfiles-v3-1-d6c2b6767b42@gmail.com > > Changes in v3: > - Only include the file list generator. Move the rest to separate patch > series. > - Remove the update-file-list make targets from img/app/Makefile and > vm/sys/net/Makefile. > - Link to v2: https://spectrum-os.org/lists/archives/spectrum-devel/20250910-genfiles-v2-0-37ebe07a3cdc@gmail.com > > Changes in v2: > - Drop the last patch (switching to /etc/s6-rc/compiled) as it is > controversial and should be reviewed separately. > - Add missing copyright notices. > - Use a wrapper shell script to make the awk code easier to read. > - Improve documentation. > - Add helper scripts for use in git hooks and rebasing. > - Link to v1: https://spectrum-os.org/lists/archives/spectrum-devel/20250903-genfiles-v1-0-cc993fcb1e4c@gmail.com/ > --- > Documentation/development/built-in-vms.adoc | 7 ++ > host/rootfs/Makefile | 102 +--------------------------- > host/rootfs/file-list.mk | 99 +++++++++++++++++++++++++++ > img/app/Makefile | 82 ++++------------------ > img/app/file-list.mk | 65 ++++++++++++++++++ > scripts/genfiles.awk | 77 +++++++++++++++++++++ > scripts/genfiles.sh | 24 +++++++ > vm/sys/net/Makefile | 52 +++----------- > vm/sys/net/file-list.mk | 42 ++++++++++++ > 9 files changed, 337 insertions(+), 213 deletions(-) > > diff --git a/host/rootfs/file-list.mk b/host/rootfs/file-list.mk > new file mode 100644 > index 0000000000000000000000000000000000000000..58cda39f85f720ab46f025bc72f1a98f108f1c25 > --- /dev/null > +++ b/host/rootfs/file-list.mk > @@ -0,0 +1,99 @@ > +# SPDX-License-Identifier: EUPL-1.2+ > +# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross Only 2021. The only thing I think is even arguably copyrightable is the comment. And following that principle let's actually put your copyright in here too. > +# Generated by scripts/genfile.sh. Any changes will be overwritten. Let's have a blank line before and after this comment, for readability. > diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk > new file mode 100644 > index 0000000000000000000000000000000000000000..935eebbdc7f0e2aa07b0b6439ab53d1f50940929 > --- /dev/null > +++ b/scripts/genfiles.awk > @@ -0,0 +1,77 @@ > +# SPDX-License-Identifier: EUPL-1.2+ > +# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross I wouldn't include this one. > +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour > +BEGIN { > + RS = "\n"; > + FS = "\t"; > + modes["120000"] = "symlink"; > + modes["100644"] = "regular"; > + modes["100755"] = "regular"; > +} > + > +function fail(msg) { > + exit_code = 1; > + print msg > "/dev/stderr"; > + exit 1; > +} > + > +# Extract data from built-in variables. > +{ > + filename = $2; > + raw_mode = $1; > + # awk autocreates empty string entries if the key is invalid, > + # but the code exits in this case so that is okay. > + mode = modes[raw_mode]; > +} > + > +filename !~ /^[[:alnum:]_./-]+$/ { > + fail("filename '" filename "' has forbidden characters"); > +} > + > +# Skip license files > +/\.license$/ { next } > + > +filename ~ /^image\/etc\/s6-rc\// { > + if (mode != "regular") { > + fail("s6-rc-compile input '" filename "' isn't a regular file"); > + } > + rc_files[rc_count++] = filename; > + next; > +} > + > +mode == "symlink" { > + symlinks[symlink_count++] = filename; > + next; > +} > + > +mode == "regular" { > + files[file_count++] = filename; > + next; > +} > + > +{ fail("File '" filename "' is not regular file or symlink (mode " raw_mode ")"); } > + > +END { > + if (exit_code) { > + exit exit_code; > + } I don't think this ever happens? > + printf ("# SPDX-License-Identifier: EUPL-1.2+\n" \ > +"# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross \n" \ > +"# Generated by scripts/genfile.sh. Any changes will be overwritten.\n" \ > +"FILES ="); > + for (array_index = 0; array_index < file_count; array_index += 1) { > + printf " \\\n\t%s", files[array_index]; > + } > + printf ("\n\n" \ > +"# These are separate because they need to be included, but putting\n" \ > +"# them as make dependencies would confuse make.\n" \ > +"LINKS ="); > + for (array_index = 0; array_index < symlink_count; array_index += 1) { > + printf " \\\n\t%s", symlinks[array_index]; > + } > + printf "\n\nS6_RC_FILES ="; > + for (array_index = 0; array_index < rc_count; array_index += 1) { > + printf " \\\n\t%s", rc_files[array_index]; > + } > + print ""; > +} --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRV/neXydHjZma5XLJbRZGEIw/wogUCaNeeIgAKCRBbRZGEIw/w onr6AQCHR/TvNsrR/uqM4oLKYGWaZHdOoPRRSTq+WzV2BMtc8wEAkvlxmyyc2Tvk y5qtQstF0mNysESLGj0LMbl68di+wQ8= =O+O4 -----END PGP SIGNATURE----- --=-=-=--