Re: [PATCH v4] Generate file lists from a script

[Alyssa Ross <hi@alyssa.is>]

[-- Attachment #1: Type: text/plain, Size: 5074 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have
> manually-maintained lists of files and symlinks.  These duplicate the
> information in the git repository and can easily get out of sync or
> cause unnecessary merge conflicts.  Fix all of these issues by having
> the git repository be the source of truth, and using a script to
> generate the file lists.  Developers can regenerate the lists before
> every commit, or even add a git hook to do that.
>
> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>

> diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk
> new file mode 100644
> index 0000000000000000000000000000000000000000..891ad162ea9748e275f7a048db3acbd9e7895a9b
> --- /dev/null
> +++ b/scripts/genfiles.awk
> @@ -0,0 +1,90 @@
> +# SPDX-License-Identifier: EUPL-1.2+
> +# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross <hi@alyssa.is>
> +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
> +BEGIN {
> +	RS = "\n";
> +	FS = "\t";
> +	modes["120000"] = "symlink";
> +	modes["100644"] = "regular";
> +	modes["100755"] = "regular";
> +}
> +
> +function fail(msg) {
> +	exit_code = 1;

This line doesn't do anything now, right?

> +	print msg > "/dev/stderr";
> +	exit 1;
> +}
> +
> +done { fail("Junk after DONE", 1); }
> +
> +$0 == "DONE" {
> +	done = 1;
> +	next;
> +}
> +
> +# Extract data from built-in variables.
> +{
> +	filename = $2;
> +	raw_mode = $1;
> +	# awk autocreates empty string entries if the key is invalid,
> +	# but the code exits in this case so that is okay.
> +	mode = modes[raw_mode];
> +}
> +
> +filename !~ /^[[:alnum:]_.+@/-]+$/ {
> +	fail("filename '" filename "' has forbidden characters");
> +}
> +
> +# Skip license files
> +/\.license$/ { next }
> +
> +filename ~ /^image\/etc\/s6-rc\// {
> +	if (mode != "regular") {
> +		fail("s6-rc-compile input '" filename "' isn't a regular file");
> +	}
> +	rc_files[rc_count++] = filename;
> +	next;
> +}
> +
> +mode == "symlink" {
> +	symlinks[symlink_count++] = filename;
> +	next;
> +}
> +
> +mode == "regular" {
> +	files[file_count++] = filename;
> +	next;
> +}
> +
> +{ fail("File '" filename "' is not regular file or symlink (mode " raw_mode ")"); }
> +
> +END {
> +	if (exit_code) {
> +		exit exit_code;
> +	}
> +	if (!done) {
> +		fail("Did not receive DONE line");
> +	}
> +	printf ("# SPDX-License-Identifier: EUPL-1.2+\n" \
> +"# SPDX-FileCopyrightText: 2021-2024 Alyssa Ross <hi@alyssa.is>\n" \
> +"# Generated by scripts/genfile.sh.  Any changes will be overwritten.\n" \
> +"FILES =") > out_file;
> +	for (array_index = 0; array_index < file_count; array_index += 1) {
> +		printf " \\\n\t%s", files[array_index] > out_file;
> +	}
> +	printf ("\n\n" \
> +"# These are separate because they need to be included, but putting\n" \
> +"# them as make dependencies would confuse make.\n" \
> +"LINKS =") > out_file;
> +	for (array_index = 0; array_index < symlink_count; array_index += 1) {
> +		printf " \\\n\t%s", symlinks[array_index] > out_file;
> +	}
> +	printf "\n\nS6_RC_FILES =" > out_file;
> +	for (array_index = 0; array_index < rc_count; array_index += 1) {
> +		printf " \\\n\t%s", rc_files[array_index] > out_file;
> +	}
> +	print > out_file;
> +	if (close(out_file)) {
> +		fail("Cannot close output file: " ERRNO);
> +	}
> +}
> diff --git a/scripts/genfiles.sh b/scripts/genfiles.sh
> new file mode 100755
> index 0000000000000000000000000000000000000000..65e8b56654448f4c9529e00807e68adb0bcfefbf
> --- /dev/null
> +++ b/scripts/genfiles.sh
> @@ -0,0 +1,28 @@
> +#!/bin/sh --
> +set -euo pipefail
> +export LC_ALL=C LANGUAGE=C
> +# shell strips trailing newlines, so add something after the newline
> +dir=$(git rev-parse --show-toplevel && echo a)
> +cd "${dir%'
> +a'}"

What's this for?  In case the directory name ends with a newline?  All
sorts of stuff is going to break if somebody decides to do this.  We
don't need to go out of our way to accomodate it.

> +for i in host/rootfs img/app vm/sys/net; do
> +    output_file=$i/file-list.mk
> +    {
> +	git -C "$i" -c core.quotePath=true ls-files $'--format=%(objectmode)\t%(path)' -- image |
> +	sort -t $'\t' -k 2
> +	echo DONE

I still don't understand what the DONE is for.  Can you describe a
circumstance in which it would be necessary?

> +    } |
> +    awk -v "out_file=$output_file.tmp" -f scripts/genfiles.awk

This was unresolved from last time too.  This could just be stdout and a
simpler awk script.  If you really want to make sure a temporary file
isn't left around if something goes wrong, you could trap EXIT, but it's
also just really not a big deal.

> +    if [ -f "$output_file" ]; then
> +	    # Avoid changing output file if it is up to date, as that
> +	    # would cause unnecessary rebuilds.
> +	    if cmp -s -- "$output_file.tmp" "$output_file"; then
> +		    rm -- "$output_file.tmp"
> +		    continue
> +	    else
> +		    astatus=$?
> +		    if [ "$astatus" != 1 ]; then exit "$astatus"; fi
> +	    fi
> +    fi
> +    mv -- "$output_file.tmp" "$output_file"
> +done

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]