Re: [PATCH v4] Generate file lists from a script

Alyssa Ross
2025-09-25T11:23:18Z

[-- Attachment #1: Type: text/plain, Size: 5074 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have
> manually-maintained lists of files and symlinks. These duplicate the
> information in the git repository and can easily get out of sync or
> cause unnecessary merge conflicts. Fix all of these issues by having
> the git repository be the source of truth, and using a script to
> generate the file lists. Developers can regenerate the lists before
> every commit, or even add a git hook to do that.
>
> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>

> diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk
> new file mode 100644
> index 0000000000000000000000000000000000000000..891ad162ea9748e275f7a048db3acbd9e7895a9b
> --- /dev/null
> +++ b/scripts/genfiles.awk
> @@ -0,0 +1,90 @@
> +# SPDX-License-Identifier: EUPL-1.2+
> +# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross <hi@alyssa.is>
> +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
> +BEGIN {
> + RS = "\n";
> + FS = "\t";
> + modes["120000"] = "symlink";
> + modes["100644"] = "regular";
> + modes["100755"] = "regular";
> +}
> +
> +function fail(msg) {
> + exit_code = 1;

This line doesn't do anything now, right?

> + print msg > "/dev/stderr";
> + exit 1;
> +}
> +
> +done { fail("Junk after DONE", 1); }
> +
> +$0 == "DONE" {
> + done = 1;
> + next;
> +}
> +
> +# Extract data from built-in variables.
> +{
> + filename = $2;
> + raw_mode = $1;
> + # awk autocreates empty string entries if the key is invalid,
> + # but the code exits in this case so that is okay.
> + mode = modes[raw_mode];
> +}
> +
> +filename !~ /^[[:alnum:]_.+@/-]+$/ {
> + fail("filename '" filename "' has forbidden characters");
> +}
> +
> +# Skip license files
> +/\.license$/ { next }
> +
> +filename ~ /^image\/etc\/s6-rc\// {
> + if (mode != "regular") {
> + fail("s6-rc-compile input '" filename "' isn't a regular file");
> + }
> + rc_files[rc_count++] = filename;
> + next;
> +}
> +
> +mode == "symlink" {
> + symlinks[symlink_count++] = filename;
> + next;
> +}
> +
> +mode == "regular" {
> + files[file_count++] = filename;
> + next;
> +}
> +
> +{ fail("File '" filename "' is not regular file or symlink (mode " raw_mode ")"); }
> +
> +END {
> + if (exit_code) {
> + exit exit_code;
> + }
> + if (!done) {
> + fail("Did not receive DONE line");
> + }
> + printf ("# SPDX-License-Identifier: EUPL-1.2+\n" \
> +"# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross <hi@alyssa.is>\n" \
> +"# Generated by scripts/genfile.sh. Any changes will be overwritten.\n" \
> +"FILES =") > out_file;
> + for (array_index = 0; array_index < file_count; array_index += 1) {
> + printf " \\\n\t%s", files[array_index] > out_file;
> + }
> + printf ("\n\n" \
> +"# These are separate because they need to be included, but putting\n" \
> +"# them as make dependencies would confuse make.\n" \
> +"LINKS =") > out_file;
> + for (array_index = 0; array_index < symlink_count; array_index += 1) {
> + printf " \\\n\t%s", symlinks[array_index] > out_file;
> + }
> + printf "\n\nS6_RC_FILES =" > out_file;
> + for (array_index = 0; array_index < rc_count; array_index += 1) {
> + printf " \\\n\t%s", rc_files[array_index] > out_file;
> + }
> + print > out_file;
> + if (close(out_file)) {
> + fail("Cannot close output file: " ERRNO);
> + }
> +}
> diff --git a/scripts/genfiles.sh b/scripts/genfiles.sh
> new file mode 100755
> index 0000000000000000000000000000000000000000..65e8b56654448f4c9529e00807e68adb0bcfefbf
> --- /dev/null
> +++ b/scripts/genfiles.sh
> @@ -0,0 +1,28 @@
> +#!/bin/sh --
> +set -euo pipefail
> +export LC_ALL=C LANGUAGE=C
> +# shell strips trailing newlines, so add something after the newline
> +dir=$(git rev-parse --show-toplevel && echo a)
> +cd "${dir%'
> +a'}"

What's this for? In case the directory name ends with a newline? All
sorts of stuff is going to break if somebody decides to do this. We
don't need to go out of our way to accomodate it.

> +for i in host/rootfs img/app vm/sys/net; do
> + output_file=$i/file-list.mk
> + {
> + git -C "$i" -c core.quotePath=true ls-files $'--format=%(objectmode)\t%(path)' -- image |
> + sort -t $'\t' -k 2
> + echo DONE

I still don't understand what the DONE is for. Can you describe a
circumstance in which it would be necessary?

> + } |
> + awk -v "out_file=$output_file.tmp" -f scripts/genfiles.awk

This was unresolved from last time too. This could just be stdout and a
simpler awk script. If you really want to make sure a temporary file
isn't left around if something goes wrong, you could trap EXIT, but it's
also just really not a big deal.

> + if [ -f "$output_file" ]; then
> + # Avoid changing output file if it is up to date, as that
> + # would cause unnecessary rebuilds.
> + if cmp -s -- "$output_file.tmp" "$output_file"; then
> + rm -- "$output_file.tmp"
> + continue
> + else
> + astatus=$?
> + if [ "$astatus" != 1 ]; then exit "$astatus"; fi
> + fi
> + fi
> + mv -- "$output_file.tmp" "$output_file"
> +done

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]