patches and low-level development discussion
 help / color / mirror / code / Atom feed
From: Demi Marie Obenour <demiobenour@gmail.com>
To: Spectrum OS Development <devel@spectrum-os.org>
Cc: Demi Marie Obenour <demiobenour@gmail.com>, Alyssa Ross <hi@alyssa.is>
Subject: [PATCH v3 3/4] Generate makefile file lists from a script
Date: Thu, 11 Sep 2025 08:47:46 -0400	[thread overview]
Message-ID: <20250911-genfiles-v3-3-d8709081dc5c@gmail.com> (raw)
In-Reply-To: <20250911-genfiles-v3-0-d8709081dc5c@gmail.com>

The script will always get them right, whereas humans (the author of
this commit included) generally will not.

Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
 Documentation/development/built-in-vms.adoc |  17 ++++
 host/rootfs/Makefile                        | 102 +----------------------
 host/rootfs/file-list.mk                    | 100 +++++++++++++++++++++++
 img/app/Makefile                            |  87 ++++----------------
 img/app/file-list.mk                        |  65 +++++++++++++++
 lib/common.mk                               |   1 +
 scripts/genfiles.awk                        | 120 ++++++++++++++++++++++++++++
 scripts/genfiles.sh                         |  23 ++++++
 scripts/git-rebase                          |  17 ++++
 scripts/pre-commit.sh                       |   9 +++
 vm/sys/net/Makefile                         |  58 ++++----------
 vm/sys/net/file-list.mk                     |  42 ++++++++++
 12 files changed, 428 insertions(+), 213 deletions(-)

diff --git a/Documentation/development/built-in-vms.adoc b/Documentation/development/built-in-vms.adoc
index e90009ee5a3c2c254a7ae11e36121576b819eee7..0addc7d1a2fd322fa12918656baa3d169478504d 100644
--- a/Documentation/development/built-in-vms.adoc
+++ b/Documentation/development/built-in-vms.adoc
@@ -44,6 +44,23 @@ NOTE: As a special convenience, it's not necessary to run `make clean`
 if the only change to the Nix files is modifying the packages
 installed in the VM.
 
+The list of files used for the VM image is stored in a separate file,
+`file-lists.mk`.  To update it, run `scripts/genfiles.sh`
+which will regenerate it from the output of `git ls-files`.  Any
+changes you made will be lost.  This script uses uses Git's index to
+generate the list, so you need to use `git add`, `git rm`, and `git mv`
+to ensure that Git knows about your changes.  It is not necessary to
+commit the changes.
+
+It is recommended that you add a Git pre-commit hook that calls
+`scripts/pre-commit.sh`.  This updates the `file-list.mk` files
+for you, so that each commit always includes the correct
+`file-list.mk`.  Sadly, Git does not run the pre-commit hook
+during rebases, so if you rebase you should use the script
+`scripts/git-rebase` instead of `git rebase`.  It tells
+Git to run `scripts/pre-commit.sh` after each commit and then
+calls `git commit --amend` to fixup the commit if needed.
+
 By default, `make run` runs the VM in Cloud Hypervisor, to match the
 Spectrum environment.  You can also add `VMM=crosvm` or `VMM=qemu` to
 your command line to run the VM in a different VMM.  One reason to do
diff --git a/host/rootfs/Makefile b/host/rootfs/Makefile
index e88a9e81e59abe461bebc9326fc861ecf47526b8..5191701c009a38062c8433a7d99288f2eaa0d1a5 100644
--- a/host/rootfs/Makefile
+++ b/host/rootfs/Makefile
@@ -4,67 +4,10 @@
 .POSIX:
 
 include ../../lib/common.mk
+include file-list.mk
 
 dest = build/rootfs.erofs
 
-FILES = \
-	image/etc/fonts/fonts.conf \
-	image/etc/fstab \
-	image/etc/group \
-	image/etc/init \
-	image/etc/login \
-	image/etc/mdev.conf \
-	image/etc/mdev/listen \
-	image/etc/mdev/net/add \
-	image/etc/mdev/wait \
-	image/etc/parse-devname \
-	image/etc/passwd \
-	image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
-	image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
-	image/etc/s6-linux-init/run-image/service/dbus/notification-fd \
-	image/etc/s6-linux-init/run-image/service/dbus/run \
-	image/etc/s6-linux-init/run-image/service/dbus/template/notification-fd \
-	image/etc/s6-linux-init/run-image/service/dbus/template/run \
-	image/etc/s6-linux-init/run-image/service/getty-tty1/run \
-	image/etc/s6-linux-init/run-image/service/getty-tty2/run \
-	image/etc/s6-linux-init/run-image/service/getty-tty3/run \
-	image/etc/s6-linux-init/run-image/service/getty-tty4/run \
-	image/etc/s6-linux-init/run-image/service/s6-svscan-log/notification-fd \
-	image/etc/s6-linux-init/run-image/service/s6-svscan-log/run \
-	image/etc/s6-linux-init/run-image/service/serial-getty-generator/run \
-	image/etc/s6-linux-init/run-image/service/serial-getty/notification-fd \
-	image/etc/s6-linux-init/run-image/service/serial-getty/run \
-	image/etc/s6-linux-init/run-image/service/serial-getty/template/run \
-	image/etc/s6-linux-init/run-image/service/vhost-user-fs/notification-fd \
-	image/etc/s6-linux-init/run-image/service/vhost-user-fs/run \
-	image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/notification-fd \
-	image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/run \
-	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/notification-fd \
-	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/run \
-	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/data/check \
-	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/notification-fd \
-	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/run \
-	image/etc/s6-linux-init/run-image/service/vmm/notification-fd \
-	image/etc/s6-linux-init/run-image/service/vmm/run \
-	image/etc/s6-linux-init/run-image/service/vmm/template/notification-fd \
-	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/notification-fd \
-	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/run \
-	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/notification-fd \
-	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/run \
-	image/etc/s6-linux-init/scripts/rc.init \
-	image/etc/xdg/weston/autolaunch \
-	image/etc/xdg/weston/weston.ini \
-	image/usr/share/dbus-1/services/org.freedesktop.portal.Documents.service \
-	image/usr/bin/assign-devices \
-	image/usr/bin/create-vm-dependencies \
-	image/usr/bin/run-appimage \
-	image/usr/bin/run-vmm \
-	image/usr/bin/vm-console \
-	image/usr/bin/vm-import \
-	image/usr/bin/vm-start \
-	image/usr/bin/vm-stop \
-	image/usr/bin/xdg-open
-
 DIRS = \
 	dev \
 	etc/s6-linux-init/env \
@@ -101,18 +44,9 @@ DIRS = \
 
 FIFOS = etc/s6-linux-init/run-image/service/s6-svscan-log/fifo
 
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-LINKS = \
-	image/bin \
-	image/etc/s6-linux-init/run-image/opengl-driver \
-	image/etc/s6-linux-init/run-image/service/vmm/template/run \
-	image/lib \
-	image/sbin
-
 BUILD_FILES = build/etc/s6-rc
 
-$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo
+$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo file-list.mk
 	( \
 	    cat $(PACKAGES_FILE) ;\
 	    for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
@@ -128,36 +62,6 @@ build/fifo:
 build/empty:
 	mkdir -p $@
 
-S6_RC_FILES = \
-	image/etc/s6-rc/card0/type \
-	image/etc/s6-rc/card0/up \
-	image/etc/s6-rc/core/type \
-	image/etc/s6-rc/core/up \
-	image/etc/s6-rc/kvm/timeout-up \
-	image/etc/s6-rc/kvm/type \
-	image/etc/s6-rc/kvm/up \
-	image/etc/s6-rc/mdevd-coldplug/dependencies \
-	image/etc/s6-rc/mdevd-coldplug/type \
-	image/etc/s6-rc/mdevd-coldplug/up \
-	image/etc/s6-rc/mdevd/notification-fd \
-	image/etc/s6-rc/mdevd/run \
-	image/etc/s6-rc/mdevd/type \
-	image/etc/s6-rc/ok-all/contents \
-	image/etc/s6-rc/ok-all/type \
-	image/etc/s6-rc/static-nodes/type \
-	image/etc/s6-rc/static-nodes/up \
-	image/etc/s6-rc/sys-vmms/dependencies \
-	image/etc/s6-rc/sys-vmms/type \
-	image/etc/s6-rc/sys-vmms/up \
-	image/etc/s6-rc/vm-env/contents \
-	image/etc/s6-rc/vm-env/type \
-	image/etc/s6-rc/vmm-env/contents \
-	image/etc/s6-rc/vmm-env/type \
-	image/etc/s6-rc/weston/dependencies \
-	image/etc/s6-rc/weston/notification-fd \
-	image/etc/s6-rc/weston/type \
-	image/etc/s6-rc/weston/run
-
 # s6-rc-compile's input is a directory, but that doesn't play nice
 # with Make, because it won't know to update if some file in the
 # directory is changed, or a file is created or removed in a
@@ -165,7 +69,7 @@ S6_RC_FILES = \
 # including files that aren't intended to be part of the input, like
 # temporary editor files or .license files.  So for all these reasons,
 # only explicitly listed files are made available to s6-rc-compile.
-build/etc/s6-rc: $(S6_RC_FILES)
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
 	mkdir -p $$(dirname $@)
 	rm -rf $@
 
diff --git a/host/rootfs/file-list.mk b/host/rootfs/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..8e5385752d5927d7c67fe4e91812f44e8c9ec033
--- /dev/null
+++ b/host/rootfs/file-list.mk
@@ -0,0 +1,100 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh.  Any changes will be overwritten.
+FILES := \
+	image/etc/fonts/fonts.conf \
+	image/etc/fstab \
+	image/etc/group \
+	image/etc/init \
+	image/etc/login \
+	image/etc/mdev.conf \
+	image/etc/mdev/listen \
+	image/etc/mdev/net/add \
+	image/etc/mdev/wait \
+	image/etc/parse-devname \
+	image/etc/passwd \
+	image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
+	image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
+	image/etc/s6-linux-init/run-image/service/dbus/notification-fd \
+	image/etc/s6-linux-init/run-image/service/dbus/run \
+	image/etc/s6-linux-init/run-image/service/dbus/template/notification-fd \
+	image/etc/s6-linux-init/run-image/service/dbus/template/run \
+	image/etc/s6-linux-init/run-image/service/getty-tty1/run \
+	image/etc/s6-linux-init/run-image/service/getty-tty2/run \
+	image/etc/s6-linux-init/run-image/service/getty-tty3/run \
+	image/etc/s6-linux-init/run-image/service/getty-tty4/run \
+	image/etc/s6-linux-init/run-image/service/s6-svscan-log/notification-fd \
+	image/etc/s6-linux-init/run-image/service/s6-svscan-log/run \
+	image/etc/s6-linux-init/run-image/service/serial-getty-generator/run \
+	image/etc/s6-linux-init/run-image/service/serial-getty/notification-fd \
+	image/etc/s6-linux-init/run-image/service/serial-getty/run \
+	image/etc/s6-linux-init/run-image/service/serial-getty/template/run \
+	image/etc/s6-linux-init/run-image/service/vhost-user-fs/notification-fd \
+	image/etc/s6-linux-init/run-image/service/vhost-user-fs/run \
+	image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/notification-fd \
+	image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/run \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/notification-fd \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/run \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/data/check \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/notification-fd \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/run \
+	image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/type \
+	image/etc/s6-linux-init/run-image/service/vmm/notification-fd \
+	image/etc/s6-linux-init/run-image/service/vmm/run \
+	image/etc/s6-linux-init/run-image/service/vmm/template/notification-fd \
+	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/notification-fd \
+	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/run \
+	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/notification-fd \
+	image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/run \
+	image/etc/s6-linux-init/scripts/rc.init \
+	image/etc/xdg/weston/autolaunch \
+	image/etc/xdg/weston/weston.ini \
+	image/usr/bin/assign-devices \
+	image/usr/bin/create-vm-dependencies \
+	image/usr/bin/run-appimage \
+	image/usr/bin/run-vmm \
+	image/usr/bin/vm-console \
+	image/usr/bin/vm-import \
+	image/usr/bin/vm-start \
+	image/usr/bin/vm-stop \
+	image/usr/bin/xdg-open \
+	image/usr/share/dbus-1/services/org.freedesktop.portal.Documents.service
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS := \
+	image/bin \
+	image/etc/s6-linux-init/run-image/opengl-driver \
+	image/etc/s6-linux-init/run-image/service/vmm/template/run \
+	image/lib \
+	image/sbin
+
+S6_RC_FILES := \
+	image/etc/s6-rc/card0/type \
+	image/etc/s6-rc/card0/up \
+	image/etc/s6-rc/core/type \
+	image/etc/s6-rc/core/up \
+	image/etc/s6-rc/kvm/timeout-up \
+	image/etc/s6-rc/kvm/type \
+	image/etc/s6-rc/kvm/up \
+	image/etc/s6-rc/mdevd-coldplug/dependencies \
+	image/etc/s6-rc/mdevd-coldplug/type \
+	image/etc/s6-rc/mdevd-coldplug/up \
+	image/etc/s6-rc/mdevd/notification-fd \
+	image/etc/s6-rc/mdevd/run \
+	image/etc/s6-rc/mdevd/type \
+	image/etc/s6-rc/ok-all/contents \
+	image/etc/s6-rc/ok-all/type \
+	image/etc/s6-rc/static-nodes/type \
+	image/etc/s6-rc/static-nodes/up \
+	image/etc/s6-rc/sys-vmms/dependencies \
+	image/etc/s6-rc/sys-vmms/type \
+	image/etc/s6-rc/sys-vmms/up \
+	image/etc/s6-rc/vm-env/contents \
+	image/etc/s6-rc/vm-env/type \
+	image/etc/s6-rc/vmm-env/contents \
+	image/etc/s6-rc/vmm-env/type \
+	image/etc/s6-rc/weston/dependencies \
+	image/etc/s6-rc/weston/notification-fd \
+	image/etc/s6-rc/weston/run \
+	image/etc/s6-rc/weston/type
diff --git a/img/app/Makefile b/img/app/Makefile
index 0099aa3b9626bfcc2079e5c67401db4e5316caeb..8be05a4781e516ee60628e2f40822cf4d6b0629e 100644
--- a/img/app/Makefile
+++ b/img/app/Makefile
@@ -5,6 +5,7 @@
 .POSIX:
 
 include ../../lib/common.mk
+include file-list.mk
 
 prefix = build/host
 libdir = $(prefix)/lib
@@ -29,45 +30,13 @@ $(imgdir)/appvm/blk/root.img: ../../scripts/make-gpt.sh ../../scripts/sfdisk-fie
 	    build/rootfs.erofs:root:5460386f-2203-4911-8694-91400125c604:root
 	mv $@.tmp $@
 
-VM_FILES = \
-	image/etc/dbus-1/session.conf \
-	image/etc/fonts/fonts.conf \
-	image/etc/fstab \
-	image/etc/init \
-	image/etc/mdev.conf \
-	image/etc/mdev/iface \
-	image/etc/mdev/listen \
-	image/etc/mdev/virtiofs \
-	image/etc/mdev/wait \
-	image/etc/passwd \
-	image/etc/pipewire/pipewire.conf \
-	image/etc/resolv.conf \
-	image/etc/s6-linux-init/env/DBUS_SESSION_BUS_ADDRESS \
-	image/etc/s6-linux-init/env/DISPLAY \
-	image/etc/s6-linux-init/env/GTK_USE_PORTAL \
-	image/etc/s6-linux-init/env/NIX_XDG_DESKTOP_PORTAL_DIR \
-	image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
-	image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
-	image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
-	image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/notification-fd \
-	image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/run \
-	image/etc/s6-linux-init/scripts/rc.init \
-	image/etc/s6-linux-init/scripts/rc.shutdown \
-	image/etc/s6-linux-init/scripts/rc.shutdown.final \
-	image/etc/wireplumber/wireplumber.conf.d/99_spectrum.conf \
-	image/etc/xdg/xdg-desktop-portal/portals.conf
-
-VM_DIRS = dev run proc sys tmp \
+DIRS = dev run proc sys tmp \
 	etc/s6-linux-init/run-image/service \
 	etc/s6-linux-init/run-image/user \
 	etc/s6-linux-init/run-image/wait
-VM_FIFOS = etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/fifo
+FIFOS = etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/fifo
 
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-VM_LINKS = image/bin image/etc/ssl/certs/ca-certificates.crt image/sbin
-
-VM_BUILD_FILES = build/etc/s6-rc
+BUILD_FILES = build/etc/s6-rc
 
 build/fifo:
 	mkdir -p build
@@ -76,49 +45,22 @@ build/fifo:
 build/empty:
 	mkdir -p $@
 
-build/rootfs.erofs: ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(VM_FILES) $(VM_BUILD_FILES) build/empty build/fifo
+build/rootfs.erofs: ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo file-list.mk
 	( \
 	    cat $(PACKAGES_FILE) ;\
-	    for file in $(VM_FILES) $(VM_LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
-	    for file in $(VM_BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
-	    printf 'build/empty\n%s\n' $(VM_DIRS) ;\
-	    printf 'build/fifo\n%s\n' $(VM_FIFOS) ;\
+	    for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file $${file#image/}; done ;\
+	    for file in $(BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
+	    printf 'build/empty\n%s\n' $(DIRS) ;\
+	    printf 'build/fifo\n%s\n' $(FIFOS) ;\
 	) | ../../scripts/make-erofs.sh $@
 
-VM_S6_RC_FILES = \
-	image/etc/s6-rc/app/dependencies.d/dbus \
-	image/etc/s6-rc/app/dependencies.d/pipewire \
-	image/etc/s6-rc/app/dependencies.d/wayland-proxy-virtwl \
-	image/etc/s6-rc/app/run \
-	image/etc/s6-rc/app/type \
-	image/etc/s6-rc/dbus/notification-fd \
-	image/etc/s6-rc/dbus/run \
-	image/etc/s6-rc/dbus/type \
-	image/etc/s6-rc/mdevd-coldplug/dependencies \
-	image/etc/s6-rc/mdevd-coldplug/type \
-	image/etc/s6-rc/mdevd-coldplug/up \
-	image/etc/s6-rc/mdevd/notification-fd \
-	image/etc/s6-rc/mdevd/run \
-	image/etc/s6-rc/mdevd/type \
-	image/etc/s6-rc/ok-all/contents \
-	image/etc/s6-rc/ok-all/type \
-	image/etc/s6-rc/pipewire/notification-fd \
-	image/etc/s6-rc/pipewire/run \
-	image/etc/s6-rc/pipewire/type \
-	image/etc/s6-rc/wayland-proxy-virtwl/notification-fd \
-	image/etc/s6-rc/wayland-proxy-virtwl/run \
-	image/etc/s6-rc/wayland-proxy-virtwl/type \
-	image/etc/s6-rc/wireplumber/dependencies.d/dbus \
-	image/etc/s6-rc/wireplumber/dependencies.d/pipewire \
-	image/etc/s6-rc/wireplumber/run \
-	image/etc/s6-rc/wireplumber/type
-
-build/etc/s6-rc: $(VM_S6_RC_FILES)
+
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
 	mkdir -p $$(dirname $@)
 	rm -rf $@
 
 	set -uo pipefail; dir=$$(mktemp -d) || exit; \
-	    { tar -c $(VM_S6_RC_FILES) | tar -C $$dir -x --strip-components 3; } && \
+	    { tar -c $(S6_RC_FILES) | tar -C $$dir -x --strip-components 3; } && \
 	    s6-rc-compile $@ $$dir; \
 	    exit=$$?; rm -r $$dir; exit $$exit
 
@@ -197,6 +139,11 @@ run-crosvm: $(imgdir)/appvm/blk/root.img start-vhost-user-gpu start-virtiofsd
 	    $(KERNEL)
 .PHONY: run-crosvm
 
+update-file-list:
+	../../scripts/genfiles.awk image > file-list.mk
+
+.PHONY: update-file-list
+
 run: run-$(VMM)
 .PHONY: run
 
diff --git a/img/app/file-list.mk b/img/app/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..2a249c27af3c5c7ab5158e3acf52400878ac6b4d
--- /dev/null
+++ b/img/app/file-list.mk
@@ -0,0 +1,65 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh.  Any changes will be overwritten.
+FILES := \
+	image/etc/dbus-1/session.conf \
+	image/etc/fonts/fonts.conf \
+	image/etc/fstab \
+	image/etc/init \
+	image/etc/mdev.conf \
+	image/etc/mdev/iface \
+	image/etc/mdev/listen \
+	image/etc/mdev/virtiofs \
+	image/etc/mdev/wait \
+	image/etc/passwd \
+	image/etc/pipewire/pipewire.conf \
+	image/etc/resolv.conf \
+	image/etc/s6-linux-init/env/DBUS_SESSION_BUS_ADDRESS \
+	image/etc/s6-linux-init/env/DISPLAY \
+	image/etc/s6-linux-init/env/GTK_USE_PORTAL \
+	image/etc/s6-linux-init/env/NIX_XDG_DESKTOP_PORTAL_DIR \
+	image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
+	image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
+	image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
+	image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/notification-fd \
+	image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/run \
+	image/etc/s6-linux-init/scripts/rc.init \
+	image/etc/s6-linux-init/scripts/rc.shutdown \
+	image/etc/s6-linux-init/scripts/rc.shutdown.final \
+	image/etc/wireplumber/wireplumber.conf.d/99_spectrum.conf \
+	image/etc/xdg/xdg-desktop-portal/portals.conf
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS := \
+	image/bin \
+	image/etc/ssl/certs/ca-certificates.crt \
+	image/sbin
+
+S6_RC_FILES := \
+	image/etc/s6-rc/app/dependencies.d/dbus \
+	image/etc/s6-rc/app/dependencies.d/pipewire \
+	image/etc/s6-rc/app/dependencies.d/wayland-proxy-virtwl \
+	image/etc/s6-rc/app/run \
+	image/etc/s6-rc/app/type \
+	image/etc/s6-rc/dbus/notification-fd \
+	image/etc/s6-rc/dbus/run \
+	image/etc/s6-rc/dbus/type \
+	image/etc/s6-rc/mdevd-coldplug/dependencies \
+	image/etc/s6-rc/mdevd-coldplug/type \
+	image/etc/s6-rc/mdevd-coldplug/up \
+	image/etc/s6-rc/mdevd/notification-fd \
+	image/etc/s6-rc/mdevd/run \
+	image/etc/s6-rc/mdevd/type \
+	image/etc/s6-rc/ok-all/contents \
+	image/etc/s6-rc/ok-all/type \
+	image/etc/s6-rc/pipewire/notification-fd \
+	image/etc/s6-rc/pipewire/run \
+	image/etc/s6-rc/pipewire/type \
+	image/etc/s6-rc/wayland-proxy-virtwl/notification-fd \
+	image/etc/s6-rc/wayland-proxy-virtwl/run \
+	image/etc/s6-rc/wayland-proxy-virtwl/type \
+	image/etc/s6-rc/wireplumber/dependencies.d/dbus \
+	image/etc/s6-rc/wireplumber/dependencies.d/pipewire \
+	image/etc/s6-rc/wireplumber/run \
+	image/etc/s6-rc/wireplumber/type
diff --git a/lib/common.mk b/lib/common.mk
index 277c3544036d9a9057f8ba4ad37fe2207548cc59..0a03ff440cc671264d2b859a2ae048db9252d047 100644
--- a/lib/common.mk
+++ b/lib/common.mk
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: EUPL-1.2+
 # SPDX-FileCopyrightText: 2021, 2023, 2025 Alyssa Ross <hi@alyssa.is>
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
 
 BACKGROUND = background
 CPIO = cpio
diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk
new file mode 100644
index 0000000000000000000000000000000000000000..88a2824c671dbf1fabca184a9e29c28d564d1cee
--- /dev/null
+++ b/scripts/genfiles.awk
@@ -0,0 +1,120 @@
+#!/usr/bin/env -S LC_ALL=C LANGUAGE=C awk -E
+# SPDX-License-Identifier: EUPL-1.2+
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+BEGIN {
+	RS = "\n";
+	FS = "\t";
+	file_count = 0;
+	symlink_count = 0;
+	rc_count = 0;
+	is_rc = 0;
+	exit_code = 0;
+	done = 0;
+	modes["120000"] = "symlink";
+	modes["040755"] = "directory";
+	modes["100644"] = "regular";
+	modes["100755"] = "regular";
+}
+
+function fail(msg, status) {
+	if (status ~ /^([1-9][0-9]?|1[0-9]{2}|2[0-4][1-9]|25[1-5])$/) {
+		exit_code = status;
+	} else {
+		exit_code = 1;
+		status = 1;
+	}
+	print ("FATAL: " msg) > "/dev/stderr";
+	exit status;
+}
+done { fail("Junk after DONE", 1); }
+/^DONE$/ {
+	done = 1
+	next
+}
+
+# Make sure git produced valid output.
+!/^[0-7]{6}\t[ -~]+$/ {
+	fail("git ls-files produced invalid output", 1);
+}
+
+# Extract data from built-in variables.
+{
+	filename = $2;
+	raw_mode = $1;
+	# awk autocreates empty string entries if the key is invalid,
+	# but the code exits in this case so that is okay.
+	mode = modes[raw_mode];
+}
+
+# Another check for a git bug.
+filename ~ /^\/|((^|\/)\.{0,2}($|\/))/ {
+	fail("git ls-files output non-canonical or absolute path '" filename "'", 1);
+}
+
+filename ~ /[^[:alnum:]_.+@/-]/ {
+	fail("filename '" filename "' has forbidden characters", 1);
+}
+
+/\.license$/ {
+	if (raw_mode != "100644") {
+		fail("License file '" filename "' is executable or not regular file", 1);
+	}
+	next;
+}
+
+mode == "directory" { next }
+
+filename ~ /^image\/etc\/s6-rc\// {
+	if (mode != "regular") {
+		fail("s6-rc-compile input '" filename "' isn't a regular file");
+	}
+	rc_count += 1;
+	rc_files[rc_count] = filename;
+	next;
+}
+
+mode == "symlink" {
+	symlink_count += 1;
+	symlinks[symlink_count] = filename;
+	next;
+}
+
+mode == "regular" {
+	file_count += 1;
+	files[file_count] = filename;
+	next;
+}
+
+{ fail("File '" filename "' is not regular file, directory, or symlink (mode " raw_mode ")"); }
+
+END {
+	if (exit_code) {
+		exit exit_code;
+	}
+	if (!done) {
+		fail("Did not receive DONE line", 1);
+	}
+	printf ("# SPDX-License-Identifier: CC0-1.0\n" \
+	       "# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>\n" \
+	       "# Generated by scripts/genfile.sh.  Any changes will be overwritten.\n" \
+	       "FILES :=") > out_file;
+	for (array_index = 1; array_index <= file_count; array_index += 1) {
+		printf " \\\n\t%s", files[array_index] > out_file;
+	}
+	printf ("\n\n" \
+"# These are separate because they need to be included, but putting\n" \
+"# them as make dependencies would confuse make.\n" \
+"LINKS :=") > out_file;
+	for (array_index = 1; array_index <= symlink_count; array_index += 1) {
+		printf " \\\n\t%s", symlinks[array_index] > out_file;
+	}
+	printf "\n\nS6_RC_FILES :=" > out_file;
+	for (array_index = 1; array_index <= rc_count; array_index += 1) {
+		printf " \\\n\t%s", rc_files[array_index] > out_file;
+	}
+	printf "\n" > out_file;
+	if (close(out_file)) {
+		print ("Cannot close output file: " ERRNO "\n") > "/dev/stderr";
+		exit 1;
+	}
+}
diff --git a/scripts/genfiles.sh b/scripts/genfiles.sh
new file mode 100755
index 0000000000000000000000000000000000000000..d90c911c98ff96e9dc2ca93103dcca902e15b09a
--- /dev/null
+++ b/scripts/genfiles.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env -S LC_ALL=C LANGUAGE=C bash --
+set -euo pipefail
+case $0 in
+(/*) cd "${0%/*}/..";;
+(*/*) cd "./${0%/*}/..";;
+(*) cd ..;;
+esac
+for i in host/rootfs img/app vm/sys/net; do
+    output_file=$i/file-list.mk
+    {
+	git -C "$i" -c core.quotePath=true ls-files $'--format=%(objectmode)\t%(path)' -- image |
+	sort -t $'\t' -k 2
+	echo DONE
+    } |
+    gawk -v "out_file=$output_file.tmp" -E scripts/genfiles.awk
+    status=0
+    cmp -s -- "$output_file.tmp" "$output_file" || status=$?
+    case $status in
+    (0) rm -- "$output_file.tmp";;
+    (1) mv -- "$output_file.tmp" "$output_file";;
+    (*) exit "$?";;
+    esac
+done
diff --git a/scripts/git-rebase b/scripts/git-rebase
new file mode 100755
index 0000000000000000000000000000000000000000..593f9299ddfd2c553ceacc68e738629ca8dec019
--- /dev/null
+++ b/scripts/git-rebase
@@ -0,0 +1,17 @@
+#!/usr/bin/env -S bash --
+set -eu
+case $0 in
+(/*) dir=${0%/*}/..;;
+(*/*) dir=./${0%/*}/..;;
+(*) dir=..;;
+esac
+exec git rebase -x "set -eu; \
+cd '${dir//\'/\'\\\'\'}'; \
+scripts/pre-commit.sh; \
+set +e;\
+git diff --quiet HEAD -- host/rootfs/gen/file-list.mk img/app/file-list.mk vm/sys/net/file-list.mk;\
+case \$? in \
+(0) :;;\
+(1) git commit --amend -- host/rootfs/gen/file-list.mk img/app/file-list.mk vm/sys/net/file-list.mk;;\
+(*) exit \"\$?\";;\
+esac" "$@"
diff --git a/scripts/pre-commit.sh b/scripts/pre-commit.sh
new file mode 100755
index 0000000000000000000000000000000000000000..29df8c0030ffeb2e611511391089de9c29763202
--- /dev/null
+++ b/scripts/pre-commit.sh
@@ -0,0 +1,9 @@
+#!/bin/sh --
+set -eu
+case $0 in
+(/*) cd ${0%/*}/..;;
+(*/*) cd ./${0%/*}/..;;
+(*) cd ..;;
+esac
+scripts/genfiles.sh
+git add -- "$i/file-list.mk"
diff --git a/vm/sys/net/Makefile b/vm/sys/net/Makefile
index b3ea49bc7065edb3411dbf5eaa3d8517ef2a61bd..23f3adcddd47a4453c4b8b7871af4516190b662d 100644
--- a/vm/sys/net/Makefile
+++ b/vm/sys/net/Makefile
@@ -4,6 +4,7 @@
 .POSIX:
 
 include ../../../lib/common.mk
+include file-list.mk
 
 prefix = build/host
 libdir = $(prefix)/lib
@@ -28,62 +29,26 @@ $(vmdir)/netvm/blk/root.img: ../../../scripts/make-gpt.sh ../../../scripts/sfdis
 	    build/rootfs.erofs:root:ea21da27-0391-48da-9235-9d2ab2ca7844:root
 	mv $@.tmp $@
 
-VM_FILES = \
-	image/etc/dbus-1/system.conf \
-	image/etc/fstab \
-	image/etc/init \
-	image/etc/mdev.conf \
-	image/etc/mdev/iface \
-	image/etc/nftables.conf \
-	image/etc/passwd \
-	image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
-	image/etc/s6-linux-init/scripts/rc.init \
-	image/etc/sysctl.conf
-VM_DIRS = dev etc/s6-linux-init/env run proc sys var/lib/connman
-
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-VM_LINKS = image/bin image/lib image/sbin image/var/run
-
-VM_BUILD_FILES = build/etc/s6-rc
+BUILD_FILES = build/etc/s6-rc
 
 build/empty:
 	mkdir -p $@
 
-build/rootfs.erofs: ../../../scripts/make-erofs.sh $(PACKAGES_FILE) $(VM_FILES) $(VM_BUILD_FILES) build/empty
+build/rootfs.erofs: ../../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty file-list.mk
 	( \
 	    cat $(PACKAGES_FILE) ;\
-	    for file in $(VM_FILES) $(VM_LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
-	    for file in $(VM_BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
-	    printf 'build/empty\n%s\n' $(VM_DIRS) ;\
+	    for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
+	    for file in $(BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
+	    printf 'build/empty\n%s\n' $(DIRS) ;\
 	) | ../../../scripts/make-erofs.sh $@
 
-VM_S6_RC_FILES = \
-	image/etc/s6-rc/connman/dependencies \
-	image/etc/s6-rc/connman/run \
-	image/etc/s6-rc/connman/type \
-	image/etc/s6-rc/dbus/notification-fd \
-	image/etc/s6-rc/dbus/run \
-	image/etc/s6-rc/dbus/type \
-	image/etc/s6-rc/mdevd-coldplug/dependencies \
-	image/etc/s6-rc/mdevd-coldplug/type \
-	image/etc/s6-rc/mdevd-coldplug/up \
-	image/etc/s6-rc/mdevd/notification-fd \
-	image/etc/s6-rc/mdevd/run \
-	image/etc/s6-rc/mdevd/type \
-	image/etc/s6-rc/nftables/type \
-	image/etc/s6-rc/nftables/up \
-	image/etc/s6-rc/ok-all/contents \
-	image/etc/s6-rc/ok-all/type \
-	image/etc/s6-rc/sysctl/type \
-	image/etc/s6-rc/sysctl/up
-
-build/etc/s6-rc: $(VM_S6_RC_FILES)
+
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
 	mkdir -p $$(dirname $@)
 	rm -rf $@
 
 	set -uo pipefail; dir=$$(mktemp -d) || exit; \
-	    { tar -c $(VM_S6_RC_FILES) | tar -C $$dir -x --strip-components 3; } && \
+	    { tar -c $(S6_RC_FILES) | tar -C $$dir -x --strip-components 3; } && \
 	    s6-rc-compile $@ $$dir; \
 	    exit=$$?; rm -r $$dir; exit $$exit
 
@@ -138,6 +103,11 @@ run-crosvm: $(vmdir)/netvm/blk/root.img
 	    $(KERNEL)
 .PHONY: run-crosvm
 
+update-file-list:
+	../../scripts/genfiles.awk image > file-list.mk
+
+.PHONY: update-file-list
+
 run: run-$(VMM)
 .PHONY: run
 
diff --git a/vm/sys/net/file-list.mk b/vm/sys/net/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..a8df7e1e0d16b6854bfe716b1134947241916119
--- /dev/null
+++ b/vm/sys/net/file-list.mk
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh.  Any changes will be overwritten.
+FILES := \
+	image/etc/dbus-1/system.conf \
+	image/etc/fstab \
+	image/etc/init \
+	image/etc/mdev.conf \
+	image/etc/mdev/iface \
+	image/etc/nftables.conf \
+	image/etc/passwd \
+	image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
+	image/etc/s6-linux-init/scripts/rc.init \
+	image/etc/sysctl.conf
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS := \
+	image/bin \
+	image/lib \
+	image/sbin \
+	image/var/run
+
+S6_RC_FILES := \
+	image/etc/s6-rc/connman/dependencies \
+	image/etc/s6-rc/connman/run \
+	image/etc/s6-rc/connman/type \
+	image/etc/s6-rc/dbus/notification-fd \
+	image/etc/s6-rc/dbus/run \
+	image/etc/s6-rc/dbus/type \
+	image/etc/s6-rc/mdevd-coldplug/dependencies \
+	image/etc/s6-rc/mdevd-coldplug/type \
+	image/etc/s6-rc/mdevd-coldplug/up \
+	image/etc/s6-rc/mdevd/notification-fd \
+	image/etc/s6-rc/mdevd/run \
+	image/etc/s6-rc/mdevd/type \
+	image/etc/s6-rc/nftables/type \
+	image/etc/s6-rc/nftables/up \
+	image/etc/s6-rc/ok-all/contents \
+	image/etc/s6-rc/ok-all/type \
+	image/etc/s6-rc/sysctl/type \
+	image/etc/s6-rc/sysctl/up

-- 
2.51.0


  parent reply	other threads:[~2025-09-11 12:51 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-09-04  1:56 [PATCH 0/4] Generate file lists from a script Demi Marie Obenour
2025-09-04  1:56 ` [PATCH 1/4] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-04  1:56 ` [PATCH 2/4] Generate makefile file lists from a script Demi Marie Obenour
2025-09-08  9:59   ` Alyssa Ross
2025-09-08 18:45     ` Demi Marie Obenour
2025-09-09 14:51       ` Alyssa Ross
2025-09-04  1:56 ` [PATCH 3/4] Common make rules for building erofs images Demi Marie Obenour
2025-09-08 10:01   ` Alyssa Ross
2025-09-08 18:53     ` Demi Marie Obenour
2025-09-09 14:56       ` Alyssa Ross
2025-09-04  1:56 ` [PATCH 4/4] Use /etc/s6-rc/compiled for compiled s6-rc directory Demi Marie Obenour
2025-09-10  5:29 ` [PATCH v2 0/3] Generate file lists from a script Demi Marie Obenour
2025-09-10  5:29   ` [PATCH v2 1/3] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-10 18:58     ` Alyssa Ross
2025-09-11 12:21       ` Demi Marie Obenour
2025-09-10  5:29   ` [PATCH v2 2/3] Generate makefile file lists from a script Demi Marie Obenour
2025-09-10  5:29   ` [PATCH v2 3/3] Common make rules for building erofs images Demi Marie Obenour
2025-09-11 12:47   ` [PATCH v3 0/4] Generate file lists from a script Demi Marie Obenour
2025-09-11 12:47     ` [PATCH v3 1/4] Do not ignore errors from tar Demi Marie Obenour
2025-09-17 11:48       ` Alyssa Ross
2025-09-18  2:45         ` Demi Marie Obenour
2025-09-19  7:46           ` Alyssa Ross
2025-09-30 12:59             ` Alyssa Ross
2025-09-19  7:55       ` Alyssa Ross
2025-09-19 19:03         ` Demi Marie Obenour
2025-09-11 12:47     ` [PATCH v3 2/4] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-17 12:30       ` Alyssa Ross
2025-09-17 12:39       ` Alyssa Ross
2025-09-17 13:03       ` Alyssa Ross
2025-09-11 12:47     ` Demi Marie Obenour [this message]
2025-09-11 12:47     ` [PATCH v3 4/4] Common make rules for building erofs images Demi Marie Obenour
2025-09-21  2:23   ` [PATCH v3] Generate file lists from a script Demi Marie Obenour
2025-09-21  8:47     ` Alyssa Ross
2025-09-21 16:51       ` Demi Marie Obenour
2025-09-21 17:07         ` Alyssa Ross
2025-09-21 17:24     ` [PATCH v4] " Demi Marie Obenour
2025-09-25 11:22       ` Alyssa Ross
2025-09-26 16:31       ` [PATCH v5] " Demi Marie Obenour
2025-09-27  8:19         ` Alyssa Ross
2025-09-27  8:42           ` Demi Marie Obenour
2025-09-27 16:22         ` [PATCH v6] " Demi Marie Obenour
2025-09-29  8:12           ` Alyssa Ross
2025-09-29 17:20             ` Demi Marie Obenour
2025-09-29 17:18           ` [PATCH v7] " Demi Marie Obenour
2025-10-01  9:20             ` Alyssa Ross
2025-10-01  9:24               ` Demi Marie Obenour
2025-10-01  9:35             ` Alyssa Ross
2025-10-01 18:30             ` [PATCH v8] " Demi Marie Obenour
2025-10-02  9:46               ` Alyssa Ross
2025-10-02 17:37               ` [PATCH v9] " Demi Marie Obenour
2025-10-03  9:04                 ` Alyssa Ross

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250911-genfiles-v3-3-d8709081dc5c@gmail.com \
    --to=demiobenour@gmail.com \
    --cc=devel@spectrum-os.org \
    --cc=hi@alyssa.is \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://spectrum-os.org/git/crosvm
	https://spectrum-os.org/git/doc
	https://spectrum-os.org/git/mktuntap
	https://spectrum-os.org/git/nixpkgs
	https://spectrum-os.org/git/spectrum
	https://spectrum-os.org/git/ucspi-vsock
	https://spectrum-os.org/git/www

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).