From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 0A06F18435; Thu, 25 Sep 2025 11:23:18 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 993) id 88A291842D; Thu, 25 Sep 2025 11:23:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-0.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_MISSING,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=4.0.1 Received: from fhigh-b1-smtp.messagingengine.com (fhigh-b1-smtp.messagingengine.com [202.12.124.152]) by atuin.qyliss.net (Postfix) with ESMTPS id 3A23A1842A for ; Thu, 25 Sep 2025 11:23:14 +0000 (UTC) Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id 484AF7A0182; Thu, 25 Sep 2025 07:23:12 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Thu, 25 Sep 2025 07:23:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1758799392; x=1758885792; bh=5K4EI2qt1D 91y5tn6SXLXj2ZzKXsJB7MTg8ZbAJeg3M=; b=gLNxggqn8ibHGtrAzZ1fzYVFVn wwUd0KENkz0kVn+5vB40gz0mI85Z8n7RUDQEjhgtaq5mfGr9jQexwvuvBTn7IUnZ 2mroy1O5FHfk03NVSD8fJdAtTes4D7QwaFercjU02bEcO4+ycyjDAJV8xA3j6Vbk HVu5DT9R/mk1pd9G+XencCWjis61sWtksGIrSqgO9UUjeFYkY8g3oLo3FRiZsCZw +0nTBzr6AcKwnO+MZKbCaYg35npq4f2axOv1MjxCH8ReMuEW4WdxOTFJjtmparxm wd1vW0xKDfMKeIcurUBjEBUoWH882PwQkRGfVlmGZA7GbndXdpqD7t+aaCzA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1758799392; x=1758885792; bh=5K4EI2qt1D91y5tn6SXLXj2ZzKXsJB7MTg8 ZbAJeg3M=; b=C7q43Y3r34rOBmTF4/Uv9CKUPbhHVek1Ja2aoIe3rRB+eqpAwM/ 8oskL5MSqph82kt8UCFN1jOK9U8l7zR9JCb/H7MhGpNBHkgi6My1/Pq+WYArzKdZ LeRTLGePAHm0iiqE7O64qHYjlr7XiHHYb2boVCIn5cLm43Cmq0pIeYblaBzV5beJ 9E53jH36eUa/Lxwv91AiUE/uuIsHho2q+BSYEzPyVLGuK+za1x84uvOgln8J5xyc XboVFlaOCT/0rQGVMkvSDCPwtCz4EZVnirdnmHYexmntsN8TGZnQDyF3nbaWGde1 6hJAecmXzuGfipc7Zy1qqaU3uIXOvBdOyrA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeiieefiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufgjfhffkfggtgesghdtreertddttdenucfhrhhomheptehlhihsshgrucft ohhsshcuoehhihesrghlhihsshgrrdhisheqnecuggftrfgrthhtvghrnhepieduffeuie elgfetgfdttddtkeekheekgfehkedufeevteegfeeiffetvdetueevnecuvehluhhsthgv rhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhephhhisegrlhihshhsrgdrih hspdhnsggprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegu vghmihhosggvnhhouhhrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepuggvvhgvlhessh hpvggtthhruhhmqdhoshdrohhrgh X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 25 Sep 2025 07:23:11 -0400 (EDT) Received: by mbp.qyliss.net (Postfix, from userid 1000) id 8C6C324D36E6; Thu, 25 Sep 2025 13:22:55 +0200 (CEST) From: Alyssa Ross To: Demi Marie Obenour Subject: Re: [PATCH v4] Generate file lists from a script In-Reply-To: <20250921-genfiles-v4-1-4375bda78707@gmail.com> References: <20250920-genfiles-v3-1-d6c2b6767b42@gmail.com> <20250921-genfiles-v4-1-4375bda78707@gmail.com> Date: Thu, 25 Sep 2025 13:22:55 +0200 Message-ID: <87tt0qbwts.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Message-ID-Hash: JY2K3SKPO6ZJ3MLY3HIFWZDMJ6WZP3CR X-Message-ID-Hash: JY2K3SKPO6ZJ3MLY3HIFWZDMJ6WZP3CR X-MailFrom: hi@alyssa.is X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; header-match-devel.spectrum-os.org-4; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Spectrum OS Development X-Mailman-Version: 3.3.9 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --=-=-= Content-Type: text/plain Demi Marie Obenour writes: > Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have > manually-maintained lists of files and symlinks. These duplicate the > information in the git repository and can easily get out of sync or > cause unnecessary merge conflicts. Fix all of these issues by having > the git repository be the source of truth, and using a script to > generate the file lists. Developers can regenerate the lists before > every commit, or even add a git hook to do that. > > Signed-off-by: Demi Marie Obenour > diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk > new file mode 100644 > index 0000000000000000000000000000000000000000..891ad162ea9748e275f7a048db3acbd9e7895a9b > --- /dev/null > +++ b/scripts/genfiles.awk > @@ -0,0 +1,90 @@ > +# SPDX-License-Identifier: EUPL-1.2+ > +# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross > +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour > +BEGIN { > + RS = "\n"; > + FS = "\t"; > + modes["120000"] = "symlink"; > + modes["100644"] = "regular"; > + modes["100755"] = "regular"; > +} > + > +function fail(msg) { > + exit_code = 1; This line doesn't do anything now, right? > + print msg > "/dev/stderr"; > + exit 1; > +} > + > +done { fail("Junk after DONE", 1); } > + > +$0 == "DONE" { > + done = 1; > + next; > +} > + > +# Extract data from built-in variables. > +{ > + filename = $2; > + raw_mode = $1; > + # awk autocreates empty string entries if the key is invalid, > + # but the code exits in this case so that is okay. > + mode = modes[raw_mode]; > +} > + > +filename !~ /^[[:alnum:]_.+@/-]+$/ { > + fail("filename '" filename "' has forbidden characters"); > +} > + > +# Skip license files > +/\.license$/ { next } > + > +filename ~ /^image\/etc\/s6-rc\// { > + if (mode != "regular") { > + fail("s6-rc-compile input '" filename "' isn't a regular file"); > + } > + rc_files[rc_count++] = filename; > + next; > +} > + > +mode == "symlink" { > + symlinks[symlink_count++] = filename; > + next; > +} > + > +mode == "regular" { > + files[file_count++] = filename; > + next; > +} > + > +{ fail("File '" filename "' is not regular file or symlink (mode " raw_mode ")"); } > + > +END { > + if (exit_code) { > + exit exit_code; > + } > + if (!done) { > + fail("Did not receive DONE line"); > + } > + printf ("# SPDX-License-Identifier: EUPL-1.2+\n" \ > +"# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross \n" \ > +"# Generated by scripts/genfile.sh. Any changes will be overwritten.\n" \ > +"FILES =") > out_file; > + for (array_index = 0; array_index < file_count; array_index += 1) { > + printf " \\\n\t%s", files[array_index] > out_file; > + } > + printf ("\n\n" \ > +"# These are separate because they need to be included, but putting\n" \ > +"# them as make dependencies would confuse make.\n" \ > +"LINKS =") > out_file; > + for (array_index = 0; array_index < symlink_count; array_index += 1) { > + printf " \\\n\t%s", symlinks[array_index] > out_file; > + } > + printf "\n\nS6_RC_FILES =" > out_file; > + for (array_index = 0; array_index < rc_count; array_index += 1) { > + printf " \\\n\t%s", rc_files[array_index] > out_file; > + } > + print > out_file; > + if (close(out_file)) { > + fail("Cannot close output file: " ERRNO); > + } > +} > diff --git a/scripts/genfiles.sh b/scripts/genfiles.sh > new file mode 100755 > index 0000000000000000000000000000000000000000..65e8b56654448f4c9529e00807e68adb0bcfefbf > --- /dev/null > +++ b/scripts/genfiles.sh > @@ -0,0 +1,28 @@ > +#!/bin/sh -- > +set -euo pipefail > +export LC_ALL=C LANGUAGE=C > +# shell strips trailing newlines, so add something after the newline > +dir=$(git rev-parse --show-toplevel && echo a) > +cd "${dir%' > +a'}" What's this for? In case the directory name ends with a newline? All sorts of stuff is going to break if somebody decides to do this. We don't need to go out of our way to accomodate it. > +for i in host/rootfs img/app vm/sys/net; do > + output_file=$i/file-list.mk > + { > + git -C "$i" -c core.quotePath=true ls-files $'--format=%(objectmode)\t%(path)' -- image | > + sort -t $'\t' -k 2 > + echo DONE I still don't understand what the DONE is for. Can you describe a circumstance in which it would be necessary? > + } | > + awk -v "out_file=$output_file.tmp" -f scripts/genfiles.awk This was unresolved from last time too. This could just be stdout and a simpler awk script. If you really want to make sure a temporary file isn't left around if something goes wrong, you could trap EXIT, but it's also just really not a big deal. > + if [ -f "$output_file" ]; then > + # Avoid changing output file if it is up to date, as that > + # would cause unnecessary rebuilds. > + if cmp -s -- "$output_file.tmp" "$output_file"; then > + rm -- "$output_file.tmp" > + continue > + else > + astatus=$? > + if [ "$astatus" != 1 ]; then exit "$astatus"; fi > + fi > + fi > + mv -- "$output_file.tmp" "$output_file" > +done --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRV/neXydHjZma5XLJbRZGEIw/wogUCaNUmDwAKCRBbRZGEIw/w og11AQDQ197iTBdPJtb0NA5VOxlUj/k586C2w1i3dpjV+hOv+QEA9zAIOqrEWWVz oDxiWawrjZrMix1Av4fkWuDOMrGPvg4= =YwoL -----END PGP SIGNATURE----- --=-=-=--