From: Demi Marie Obenour <demiobenour@gmail.com>
To: Spectrum OS Development <devel@spectrum-os.org>
Cc: Alyssa Ross <hi@alyssa.is>, Demi Marie Obenour <demiobenour@gmail.com>
Subject: [PATCH v3] Generate file lists from a script
Date: Sat, 20 Sep 2025 22:23:44 -0400 [thread overview]
Message-ID: <20250920-genfiles-v3-1-d6c2b6767b42@gmail.com> (raw)
In-Reply-To: <20250910-genfiles-v2-0-37ebe07a3cdc@gmail.com>
Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have
manually-maintained lists of files and symlinks. These duplicate the
information in the git repository and can easily get out of sync or
cause unnecessary merge conflicts. Fix all of these issues by having
the git repository be the source of truth, and using a script to
generate the file lists. Developers can regenerate the lists before
every commit, or even add a git hook to do that.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
Changes in v3:
- Only include the file list generator. Move the rest to separate patch
series.
- Remove the update-file-list make targets from img/app/Makefile and
vm/sys/net/Makefile.
- Link to v2: https://lore.kernel.org/r/20250910-genfiles-v2-0-37ebe07a3cdc@gmail.com
Changes in v2:
- Drop the last patch (switching to /etc/s6-rc/compiled) as it is
controversial and should be reviewed separately.
- Add missing copyright notices.
- Use a wrapper shell script to make the awk code easier to read.
- Improve documentation.
- Add helper scripts for use in git hooks and rebasing.
- Link to v1: https://spectrum-os.org/lists/archives/spectrum-devel/20250903-genfiles-v1-0-cc993fcb1e4c@gmail.com/
---
Documentation/development/built-in-vms.adoc | 17 ++++
host/rootfs/Makefile | 102 +----------------------
host/rootfs/file-list.mk | 99 +++++++++++++++++++++++
img/app/Makefile | 80 +++----------------
img/app/file-list.mk | 65 +++++++++++++++
lib/common.mk | 1 +
scripts/genfiles.awk | 120 ++++++++++++++++++++++++++++
scripts/genfiles.sh | 29 +++++++
scripts/git-rebase | 17 ++++
scripts/pre-commit.sh | 11 +++
vm/sys/net/Makefile | 50 ++----------
vm/sys/net/file-list.mk | 42 ++++++++++
12 files changed, 422 insertions(+), 211 deletions(-)
diff --git a/Documentation/development/built-in-vms.adoc b/Documentation/development/built-in-vms.adoc
index e90009ee5a3c2c254a7ae11e36121576b819eee7..0addc7d1a2fd322fa12918656baa3d169478504d 100644
--- a/Documentation/development/built-in-vms.adoc
+++ b/Documentation/development/built-in-vms.adoc
@@ -44,6 +44,23 @@ NOTE: As a special convenience, it's not necessary to run `make clean`
if the only change to the Nix files is modifying the packages
installed in the VM.
+The list of files used for the VM image is stored in a separate file,
+`file-lists.mk`. To update it, run `scripts/genfiles.sh`
+which will regenerate it from the output of `git ls-files`. Any
+changes you made will be lost. This script uses uses Git's index to
+generate the list, so you need to use `git add`, `git rm`, and `git mv`
+to ensure that Git knows about your changes. It is not necessary to
+commit the changes.
+
+It is recommended that you add a Git pre-commit hook that calls
+`scripts/pre-commit.sh`. This updates the `file-list.mk` files
+for you, so that each commit always includes the correct
+`file-list.mk`. Sadly, Git does not run the pre-commit hook
+during rebases, so if you rebase you should use the script
+`scripts/git-rebase` instead of `git rebase`. It tells
+Git to run `scripts/pre-commit.sh` after each commit and then
+calls `git commit --amend` to fixup the commit if needed.
+
By default, `make run` runs the VM in Cloud Hypervisor, to match the
Spectrum environment. You can also add `VMM=crosvm` or `VMM=qemu` to
your command line to run the VM in a different VMM. One reason to do
diff --git a/host/rootfs/Makefile b/host/rootfs/Makefile
index b11ba50f369c3440572e06a7eaca429fbe6746a1..875e511e6a5a83346fbf507d1104ce70599f0e21 100644
--- a/host/rootfs/Makefile
+++ b/host/rootfs/Makefile
@@ -4,67 +4,10 @@
.POSIX:
include ../../lib/common.mk
+include file-list.mk
dest = build/rootfs.erofs
-FILES = \
- image/etc/fonts/fonts.conf \
- image/etc/fstab \
- image/etc/group \
- image/etc/init \
- image/etc/login \
- image/etc/mdev.conf \
- image/etc/mdev/listen \
- image/etc/mdev/net/add \
- image/etc/mdev/wait \
- image/etc/parse-devname \
- image/etc/passwd \
- image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
- image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
- image/etc/s6-linux-init/run-image/service/dbus/notification-fd \
- image/etc/s6-linux-init/run-image/service/dbus/run \
- image/etc/s6-linux-init/run-image/service/dbus/template/notification-fd \
- image/etc/s6-linux-init/run-image/service/dbus/template/run \
- image/etc/s6-linux-init/run-image/service/getty-tty1/run \
- image/etc/s6-linux-init/run-image/service/getty-tty2/run \
- image/etc/s6-linux-init/run-image/service/getty-tty3/run \
- image/etc/s6-linux-init/run-image/service/getty-tty4/run \
- image/etc/s6-linux-init/run-image/service/s6-svscan-log/notification-fd \
- image/etc/s6-linux-init/run-image/service/s6-svscan-log/run \
- image/etc/s6-linux-init/run-image/service/serial-getty-generator/run \
- image/etc/s6-linux-init/run-image/service/serial-getty/notification-fd \
- image/etc/s6-linux-init/run-image/service/serial-getty/run \
- image/etc/s6-linux-init/run-image/service/serial-getty/template/run \
- image/etc/s6-linux-init/run-image/service/vhost-user-fs/notification-fd \
- image/etc/s6-linux-init/run-image/service/vhost-user-fs/run \
- image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/notification-fd \
- image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/run \
- image/etc/s6-linux-init/run-image/service/vhost-user-gpu/notification-fd \
- image/etc/s6-linux-init/run-image/service/vhost-user-gpu/run \
- image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/data/check \
- image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/notification-fd \
- image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/run \
- image/etc/s6-linux-init/run-image/service/vmm/notification-fd \
- image/etc/s6-linux-init/run-image/service/vmm/run \
- image/etc/s6-linux-init/run-image/service/vmm/template/notification-fd \
- image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/notification-fd \
- image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/run \
- image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/notification-fd \
- image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/run \
- image/etc/s6-linux-init/scripts/rc.init \
- image/etc/xdg/weston/autolaunch \
- image/etc/xdg/weston/weston.ini \
- image/usr/share/dbus-1/services/org.freedesktop.portal.Documents.service \
- image/usr/bin/assign-devices \
- image/usr/bin/create-vm-dependencies \
- image/usr/bin/run-appimage \
- image/usr/bin/run-vmm \
- image/usr/bin/vm-console \
- image/usr/bin/vm-import \
- image/usr/bin/vm-start \
- image/usr/bin/vm-stop \
- image/usr/bin/xdg-open
-
DIRS = \
dev \
etc/s6-linux-init/env \
@@ -101,18 +44,9 @@ DIRS = \
FIFOS = etc/s6-linux-init/run-image/service/s6-svscan-log/fifo
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-LINKS = \
- image/bin \
- image/etc/s6-linux-init/run-image/opengl-driver \
- image/etc/s6-linux-init/run-image/service/vmm/template/run \
- image/lib \
- image/sbin
-
BUILD_FILES = build/etc/s6-rc
-$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo
+$(dest): ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo file-list.mk
( \
cat $(PACKAGES_FILE) ;\
for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
@@ -128,36 +62,6 @@ build/fifo:
build/empty:
mkdir -p $@
-S6_RC_FILES = \
- image/etc/s6-rc/card0/type \
- image/etc/s6-rc/card0/up \
- image/etc/s6-rc/core/type \
- image/etc/s6-rc/core/up \
- image/etc/s6-rc/kvm/timeout-up \
- image/etc/s6-rc/kvm/type \
- image/etc/s6-rc/kvm/up \
- image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
- image/etc/s6-rc/mdevd-coldplug/type \
- image/etc/s6-rc/mdevd-coldplug/up \
- image/etc/s6-rc/mdevd/notification-fd \
- image/etc/s6-rc/mdevd/run \
- image/etc/s6-rc/mdevd/type \
- image/etc/s6-rc/ok-all/contents \
- image/etc/s6-rc/ok-all/type \
- image/etc/s6-rc/static-nodes/type \
- image/etc/s6-rc/static-nodes/up \
- image/etc/s6-rc/sys-vmms/dependencies.d/vmm-env \
- image/etc/s6-rc/sys-vmms/type \
- image/etc/s6-rc/sys-vmms/up \
- image/etc/s6-rc/vm-env/contents \
- image/etc/s6-rc/vm-env/type \
- image/etc/s6-rc/vmm-env/contents \
- image/etc/s6-rc/vmm-env/type \
- image/etc/s6-rc/weston/dependencies.d/card0 \
- image/etc/s6-rc/weston/notification-fd \
- image/etc/s6-rc/weston/run \
- image/etc/s6-rc/weston/type
-
# s6-rc-compile's input is a directory, but that doesn't play nice
# with Make, because it won't know to update if some file in the
# directory is changed, or a file is created or removed in a
@@ -165,7 +69,7 @@ S6_RC_FILES = \
# including files that aren't intended to be part of the input, like
# temporary editor files or .license files. So for all these reasons,
# only explicitly listed files are made available to s6-rc-compile.
-build/etc/s6-rc: $(S6_RC_FILES)
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
mkdir -p $$(dirname $@)
rm -rf $@
diff --git a/host/rootfs/file-list.mk b/host/rootfs/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..f9946a6fa10e0cdf7d32975c3cc6d4b9e0d38991
--- /dev/null
+++ b/host/rootfs/file-list.mk
@@ -0,0 +1,99 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh. Any changes will be overwritten.
+FILES ::= \
+ image/etc/fonts/fonts.conf \
+ image/etc/fstab \
+ image/etc/group \
+ image/etc/init \
+ image/etc/login \
+ image/etc/mdev.conf \
+ image/etc/mdev/listen \
+ image/etc/mdev/net/add \
+ image/etc/mdev/wait \
+ image/etc/parse-devname \
+ image/etc/passwd \
+ image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
+ image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
+ image/etc/s6-linux-init/run-image/service/dbus/notification-fd \
+ image/etc/s6-linux-init/run-image/service/dbus/run \
+ image/etc/s6-linux-init/run-image/service/dbus/template/notification-fd \
+ image/etc/s6-linux-init/run-image/service/dbus/template/run \
+ image/etc/s6-linux-init/run-image/service/getty-tty1/run \
+ image/etc/s6-linux-init/run-image/service/getty-tty2/run \
+ image/etc/s6-linux-init/run-image/service/getty-tty3/run \
+ image/etc/s6-linux-init/run-image/service/getty-tty4/run \
+ image/etc/s6-linux-init/run-image/service/s6-svscan-log/notification-fd \
+ image/etc/s6-linux-init/run-image/service/s6-svscan-log/run \
+ image/etc/s6-linux-init/run-image/service/serial-getty-generator/run \
+ image/etc/s6-linux-init/run-image/service/serial-getty/notification-fd \
+ image/etc/s6-linux-init/run-image/service/serial-getty/run \
+ image/etc/s6-linux-init/run-image/service/serial-getty/template/run \
+ image/etc/s6-linux-init/run-image/service/vhost-user-fs/notification-fd \
+ image/etc/s6-linux-init/run-image/service/vhost-user-fs/run \
+ image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/notification-fd \
+ image/etc/s6-linux-init/run-image/service/vhost-user-fs/template/run \
+ image/etc/s6-linux-init/run-image/service/vhost-user-gpu/notification-fd \
+ image/etc/s6-linux-init/run-image/service/vhost-user-gpu/run \
+ image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/data/check \
+ image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/notification-fd \
+ image/etc/s6-linux-init/run-image/service/vhost-user-gpu/template/run \
+ image/etc/s6-linux-init/run-image/service/vmm/notification-fd \
+ image/etc/s6-linux-init/run-image/service/vmm/run \
+ image/etc/s6-linux-init/run-image/service/vmm/template/notification-fd \
+ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/notification-fd \
+ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/run \
+ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/notification-fd \
+ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/run \
+ image/etc/s6-linux-init/scripts/rc.init \
+ image/etc/xdg/weston/autolaunch \
+ image/etc/xdg/weston/weston.ini \
+ image/usr/bin/assign-devices \
+ image/usr/bin/create-vm-dependencies \
+ image/usr/bin/run-appimage \
+ image/usr/bin/run-vmm \
+ image/usr/bin/vm-console \
+ image/usr/bin/vm-import \
+ image/usr/bin/vm-start \
+ image/usr/bin/vm-stop \
+ image/usr/bin/xdg-open \
+ image/usr/share/dbus-1/services/org.freedesktop.portal.Documents.service
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS ::= \
+ image/bin \
+ image/etc/s6-linux-init/run-image/opengl-driver \
+ image/etc/s6-linux-init/run-image/service/vmm/template/run \
+ image/lib \
+ image/sbin
+
+S6_RC_FILES ::= \
+ image/etc/s6-rc/card0/type \
+ image/etc/s6-rc/card0/up \
+ image/etc/s6-rc/core/type \
+ image/etc/s6-rc/core/up \
+ image/etc/s6-rc/kvm/timeout-up \
+ image/etc/s6-rc/kvm/type \
+ image/etc/s6-rc/kvm/up \
+ image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
+ image/etc/s6-rc/mdevd-coldplug/type \
+ image/etc/s6-rc/mdevd-coldplug/up \
+ image/etc/s6-rc/mdevd/notification-fd \
+ image/etc/s6-rc/mdevd/run \
+ image/etc/s6-rc/mdevd/type \
+ image/etc/s6-rc/ok-all/contents \
+ image/etc/s6-rc/ok-all/type \
+ image/etc/s6-rc/static-nodes/type \
+ image/etc/s6-rc/static-nodes/up \
+ image/etc/s6-rc/sys-vmms/dependencies.d/vmm-env \
+ image/etc/s6-rc/sys-vmms/type \
+ image/etc/s6-rc/sys-vmms/up \
+ image/etc/s6-rc/vm-env/contents \
+ image/etc/s6-rc/vm-env/type \
+ image/etc/s6-rc/vmm-env/contents \
+ image/etc/s6-rc/vmm-env/type \
+ image/etc/s6-rc/weston/dependencies.d/card0 \
+ image/etc/s6-rc/weston/notification-fd \
+ image/etc/s6-rc/weston/run \
+ image/etc/s6-rc/weston/type
diff --git a/img/app/Makefile b/img/app/Makefile
index c83b37364c16576e2609f5005b8a6092b644867b..61f783a1987791aef0bf062a78a44e1371fca80f 100644
--- a/img/app/Makefile
+++ b/img/app/Makefile
@@ -5,6 +5,7 @@
.POSIX:
include ../../lib/common.mk
+include file-list.mk
prefix = build/host
libdir = $(prefix)/lib
@@ -29,45 +30,13 @@ $(imgdir)/appvm/blk/root.img: ../../scripts/make-gpt.sh ../../scripts/sfdisk-fie
build/rootfs.erofs:root:5460386f-2203-4911-8694-91400125c604:root
mv $@.tmp $@
-VM_FILES = \
- image/etc/dbus-1/session.conf \
- image/etc/fonts/fonts.conf \
- image/etc/fstab \
- image/etc/init \
- image/etc/mdev.conf \
- image/etc/mdev/iface \
- image/etc/mdev/listen \
- image/etc/mdev/virtiofs \
- image/etc/mdev/wait \
- image/etc/passwd \
- image/etc/pipewire/pipewire.conf \
- image/etc/resolv.conf \
- image/etc/s6-linux-init/env/DBUS_SESSION_BUS_ADDRESS \
- image/etc/s6-linux-init/env/DISPLAY \
- image/etc/s6-linux-init/env/GTK_USE_PORTAL \
- image/etc/s6-linux-init/env/NIX_XDG_DESKTOP_PORTAL_DIR \
- image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
- image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
- image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
- image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/notification-fd \
- image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/run \
- image/etc/s6-linux-init/scripts/rc.init \
- image/etc/s6-linux-init/scripts/rc.shutdown \
- image/etc/s6-linux-init/scripts/rc.shutdown.final \
- image/etc/wireplumber/wireplumber.conf.d/99_spectrum.conf \
- image/etc/xdg/xdg-desktop-portal/portals.conf
-
-VM_DIRS = dev run proc sys tmp \
+DIRS = dev run proc sys tmp \
etc/s6-linux-init/run-image/service \
etc/s6-linux-init/run-image/user \
etc/s6-linux-init/run-image/wait
-VM_FIFOS = etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/fifo
+FIFOS = etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/fifo
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-VM_LINKS = image/bin image/etc/ssl/certs/ca-certificates.crt image/sbin
-
-VM_BUILD_FILES = build/etc/s6-rc
+BUILD_FILES = build/etc/s6-rc
build/fifo:
mkdir -p build
@@ -76,44 +45,17 @@ build/fifo:
build/empty:
mkdir -p $@
-build/rootfs.erofs: ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(VM_FILES) $(VM_BUILD_FILES) build/empty build/fifo
+build/rootfs.erofs: ../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty build/fifo file-list.mk
( \
cat $(PACKAGES_FILE) ;\
- for file in $(VM_FILES) $(VM_LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
- for file in $(VM_BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
- printf 'build/empty\n%s\n' $(VM_DIRS) ;\
- printf 'build/fifo\n%s\n' $(VM_FIFOS) ;\
+ for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file $${file#image/}; done ;\
+ for file in $(BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
+ printf 'build/empty\n%s\n' $(DIRS) ;\
+ printf 'build/fifo\n%s\n' $(FIFOS) ;\
) | ../../scripts/make-erofs.sh $@
-VM_S6_RC_FILES = \
- image/etc/s6-rc/app/dependencies.d/dbus \
- image/etc/s6-rc/app/dependencies.d/pipewire \
- image/etc/s6-rc/app/dependencies.d/wayland-proxy-virtwl \
- image/etc/s6-rc/app/run \
- image/etc/s6-rc/app/type \
- image/etc/s6-rc/dbus/notification-fd \
- image/etc/s6-rc/dbus/run \
- image/etc/s6-rc/dbus/type \
- image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
- image/etc/s6-rc/mdevd-coldplug/type \
- image/etc/s6-rc/mdevd-coldplug/up \
- image/etc/s6-rc/mdevd/notification-fd \
- image/etc/s6-rc/mdevd/run \
- image/etc/s6-rc/mdevd/type \
- image/etc/s6-rc/ok-all/contents \
- image/etc/s6-rc/ok-all/type \
- image/etc/s6-rc/pipewire/notification-fd \
- image/etc/s6-rc/pipewire/run \
- image/etc/s6-rc/pipewire/type \
- image/etc/s6-rc/wayland-proxy-virtwl/notification-fd \
- image/etc/s6-rc/wayland-proxy-virtwl/run \
- image/etc/s6-rc/wayland-proxy-virtwl/type \
- image/etc/s6-rc/wireplumber/dependencies.d/dbus \
- image/etc/s6-rc/wireplumber/dependencies.d/pipewire \
- image/etc/s6-rc/wireplumber/run \
- image/etc/s6-rc/wireplumber/type
-
-build/etc/s6-rc: $(VM_S6_RC_FILES)
+
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
mkdir -p $$(dirname $@)
rm -rf $@
diff --git a/img/app/file-list.mk b/img/app/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..049d597e22536d64d30d4af6e220d0a169b9b924
--- /dev/null
+++ b/img/app/file-list.mk
@@ -0,0 +1,65 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh. Any changes will be overwritten.
+FILES ::= \
+ image/etc/dbus-1/session.conf \
+ image/etc/fonts/fonts.conf \
+ image/etc/fstab \
+ image/etc/init \
+ image/etc/mdev.conf \
+ image/etc/mdev/iface \
+ image/etc/mdev/listen \
+ image/etc/mdev/virtiofs \
+ image/etc/mdev/wait \
+ image/etc/passwd \
+ image/etc/pipewire/pipewire.conf \
+ image/etc/resolv.conf \
+ image/etc/s6-linux-init/env/DBUS_SESSION_BUS_ADDRESS \
+ image/etc/s6-linux-init/env/DISPLAY \
+ image/etc/s6-linux-init/env/GTK_USE_PORTAL \
+ image/etc/s6-linux-init/env/NIX_XDG_DESKTOP_PORTAL_DIR \
+ image/etc/s6-linux-init/env/WAYLAND_DISPLAY \
+ image/etc/s6-linux-init/env/XDG_RUNTIME_DIR \
+ image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
+ image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/notification-fd \
+ image/etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/run \
+ image/etc/s6-linux-init/scripts/rc.init \
+ image/etc/s6-linux-init/scripts/rc.shutdown \
+ image/etc/s6-linux-init/scripts/rc.shutdown.final \
+ image/etc/wireplumber/wireplumber.conf.d/99_spectrum.conf \
+ image/etc/xdg/xdg-desktop-portal/portals.conf
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS ::= \
+ image/bin \
+ image/etc/ssl/certs/ca-certificates.crt \
+ image/sbin
+
+S6_RC_FILES ::= \
+ image/etc/s6-rc/app/dependencies.d/dbus \
+ image/etc/s6-rc/app/dependencies.d/pipewire \
+ image/etc/s6-rc/app/dependencies.d/wayland-proxy-virtwl \
+ image/etc/s6-rc/app/run \
+ image/etc/s6-rc/app/type \
+ image/etc/s6-rc/dbus/notification-fd \
+ image/etc/s6-rc/dbus/run \
+ image/etc/s6-rc/dbus/type \
+ image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
+ image/etc/s6-rc/mdevd-coldplug/type \
+ image/etc/s6-rc/mdevd-coldplug/up \
+ image/etc/s6-rc/mdevd/notification-fd \
+ image/etc/s6-rc/mdevd/run \
+ image/etc/s6-rc/mdevd/type \
+ image/etc/s6-rc/ok-all/contents \
+ image/etc/s6-rc/ok-all/type \
+ image/etc/s6-rc/pipewire/notification-fd \
+ image/etc/s6-rc/pipewire/run \
+ image/etc/s6-rc/pipewire/type \
+ image/etc/s6-rc/wayland-proxy-virtwl/notification-fd \
+ image/etc/s6-rc/wayland-proxy-virtwl/run \
+ image/etc/s6-rc/wayland-proxy-virtwl/type \
+ image/etc/s6-rc/wireplumber/dependencies.d/dbus \
+ image/etc/s6-rc/wireplumber/dependencies.d/pipewire \
+ image/etc/s6-rc/wireplumber/run \
+ image/etc/s6-rc/wireplumber/type
diff --git a/lib/common.mk b/lib/common.mk
index 277c3544036d9a9057f8ba4ad37fe2207548cc59..0a03ff440cc671264d2b859a2ae048db9252d047 100644
--- a/lib/common.mk
+++ b/lib/common.mk
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: EUPL-1.2+
# SPDX-FileCopyrightText: 2021, 2023, 2025 Alyssa Ross <hi@alyssa.is>
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
BACKGROUND = background
CPIO = cpio
diff --git a/scripts/genfiles.awk b/scripts/genfiles.awk
new file mode 100644
index 0000000000000000000000000000000000000000..6fe327fd0a314d226dbce23854aa8f119e9c8f34
--- /dev/null
+++ b/scripts/genfiles.awk
@@ -0,0 +1,120 @@
+#!/usr/bin/env -S LC_ALL=C LANGUAGE=C awk -E
+# SPDX-License-Identifier: EUPL-1.2+
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+BEGIN {
+ RS = "\n";
+ FS = "\t";
+ file_count = 0;
+ symlink_count = 0;
+ rc_count = 0;
+ is_rc = 0;
+ exit_code = 0;
+ done = 0;
+ modes["120000"] = "symlink";
+ modes["040755"] = "directory";
+ modes["100644"] = "regular";
+ modes["100755"] = "regular";
+}
+
+function fail(msg, status) {
+ if (status ~ /^([1-9][0-9]?|1[0-9]{2}|2[0-4][1-9]|25[1-5])$/) {
+ exit_code = status;
+ } else {
+ exit_code = 1;
+ status = 1;
+ }
+ print ("FATAL: " msg) > "/dev/stderr";
+ exit status;
+}
+done { fail("Junk after DONE", 1); }
+/^DONE$/ {
+ done = 1
+ next
+}
+
+# Make sure git produced valid output.
+!/^[0-7]{6}\t[ -~]+$/ {
+ fail("git ls-files produced invalid output", 1);
+}
+
+# Extract data from built-in variables.
+{
+ filename = $2;
+ raw_mode = $1;
+ # awk autocreates empty string entries if the key is invalid,
+ # but the code exits in this case so that is okay.
+ mode = modes[raw_mode];
+}
+
+# Another check for a git bug.
+filename ~ /^\/|((^|\/)\.{0,2}($|\/))/ {
+ fail("git ls-files output non-canonical or absolute path '" filename "'", 1);
+}
+
+filename ~ /[^[:alnum:]_.+@/-]/ {
+ fail("filename '" filename "' has forbidden characters", 1);
+}
+
+/\.license$/ {
+ if (raw_mode != "100644") {
+ fail("License file '" filename "' is executable or not regular file", 1);
+ }
+ next;
+}
+
+mode == "directory" { next }
+
+filename ~ /^image\/etc\/s6-rc\// {
+ if (mode != "regular") {
+ fail("s6-rc-compile input '" filename "' isn't a regular file");
+ }
+ rc_count += 1;
+ rc_files[rc_count] = filename;
+ next;
+}
+
+mode == "symlink" {
+ symlink_count += 1;
+ symlinks[symlink_count] = filename;
+ next;
+}
+
+mode == "regular" {
+ file_count += 1;
+ files[file_count] = filename;
+ next;
+}
+
+{ fail("File '" filename "' is not regular file, directory, or symlink (mode " raw_mode ")"); }
+
+END {
+ if (exit_code) {
+ exit exit_code;
+ }
+ if (!done) {
+ fail("Did not receive DONE line", 1);
+ }
+ printf ("# SPDX-License-Identifier: CC0-1.0\n" \
+ "# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>\n" \
+ "# Generated by scripts/genfile.sh. Any changes will be overwritten.\n" \
+ "FILES ::=") > out_file;
+ for (array_index = 1; array_index <= file_count; array_index += 1) {
+ printf " \\\n\t%s", files[array_index] > out_file;
+ }
+ printf ("\n\n" \
+"# These are separate because they need to be included, but putting\n" \
+"# them as make dependencies would confuse make.\n" \
+"LINKS ::=") > out_file;
+ for (array_index = 1; array_index <= symlink_count; array_index += 1) {
+ printf " \\\n\t%s", symlinks[array_index] > out_file;
+ }
+ printf "\n\nS6_RC_FILES ::=" > out_file;
+ for (array_index = 1; array_index <= rc_count; array_index += 1) {
+ printf " \\\n\t%s", rc_files[array_index] > out_file;
+ }
+ printf "\n" > out_file;
+ if (close(out_file)) {
+ print ("Cannot close output file: " ERRNO "\n") > "/dev/stderr";
+ exit 1;
+ }
+}
diff --git a/scripts/genfiles.sh b/scripts/genfiles.sh
new file mode 100755
index 0000000000000000000000000000000000000000..77a8d95e88b6851be9447698556efe4f1eab174b
--- /dev/null
+++ b/scripts/genfiles.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env -S LC_ALL=C LANGUAGE=C bash --
+set -euo pipefail
+unset output_file astatus
+case $0 in
+(/*) cd "${0%/*}/..";;
+(*/*) cd "./${0%/*}/..";;
+(*) cd ..;;
+esac
+for i in host/rootfs img/app vm/sys/net; do
+ output_file=$i/file-list.mk
+ {
+ git -C "$i" -c core.quotePath=true ls-files $'--format=%(objectmode)\t%(path)' -- image |
+ sort -t $'\t' -k 2
+ echo DONE
+ } |
+ gawk -v "out_file=$output_file.tmp" -E scripts/genfiles.awk
+ if [ -f "$output_file" ]; then
+ # Avoid changing output file if it is up to date, as that
+ # would cause unnecessary rebuilds.
+ if cmp -s -- "$output_file.tmp" "$output_file"; then
+ rm -- "$output_file.tmp"
+ continue
+ else
+ astatus=$?
+ if [ "$astatus" != 1 ]; then exit "$astatus"; fi
+ fi
+ fi
+ mv -- "$output_file.tmp" "$output_file"
+done
diff --git a/scripts/git-rebase b/scripts/git-rebase
new file mode 100755
index 0000000000000000000000000000000000000000..593f9299ddfd2c553ceacc68e738629ca8dec019
--- /dev/null
+++ b/scripts/git-rebase
@@ -0,0 +1,17 @@
+#!/usr/bin/env -S bash --
+set -eu
+case $0 in
+(/*) dir=${0%/*}/..;;
+(*/*) dir=./${0%/*}/..;;
+(*) dir=..;;
+esac
+exec git rebase -x "set -eu; \
+cd '${dir//\'/\'\\\'\'}'; \
+scripts/pre-commit.sh; \
+set +e;\
+git diff --quiet HEAD -- host/rootfs/gen/file-list.mk img/app/file-list.mk vm/sys/net/file-list.mk;\
+case \$? in \
+(0) :;;\
+(1) git commit --amend -- host/rootfs/gen/file-list.mk img/app/file-list.mk vm/sys/net/file-list.mk;;\
+(*) exit \"\$?\";;\
+esac" "$@"
diff --git a/scripts/pre-commit.sh b/scripts/pre-commit.sh
new file mode 100755
index 0000000000000000000000000000000000000000..f0858da5395954e5af86a1fc4865c649381e5fdd
--- /dev/null
+++ b/scripts/pre-commit.sh
@@ -0,0 +1,11 @@
+#!/bin/sh --
+set -eu
+case $0 in
+(/*) cd ${0%/*}/..;;
+(*/*) cd ./${0%/*}/..;;
+(*) cd ..;;
+esac
+scripts/genfiles.sh
+for i in host/rootfs img/app vm/sys/net; do
+ git add -- "$i/file-list.mk"
+done
diff --git a/vm/sys/net/Makefile b/vm/sys/net/Makefile
index 79be242ea26e37d6f02af8761320de27d69d5b69..e0a3c34b584569bc196937753c3d7669ca60ab3d 100644
--- a/vm/sys/net/Makefile
+++ b/vm/sys/net/Makefile
@@ -4,6 +4,7 @@
.POSIX:
include ../../../lib/common.mk
+include file-list.mk
prefix = build/host
libdir = $(prefix)/lib
@@ -28,57 +29,20 @@ $(vmdir)/netvm/blk/root.img: ../../../scripts/make-gpt.sh ../../../scripts/sfdis
build/rootfs.erofs:root:ea21da27-0391-48da-9235-9d2ab2ca7844:root
mv $@.tmp $@
-VM_FILES = \
- image/etc/dbus-1/system.conf \
- image/etc/fstab \
- image/etc/init \
- image/etc/mdev.conf \
- image/etc/mdev/iface \
- image/etc/nftables.conf \
- image/etc/passwd \
- image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
- image/etc/s6-linux-init/scripts/rc.init \
- image/etc/sysctl.conf
-VM_DIRS = dev etc/s6-linux-init/env run proc sys var/lib/connman
-
-# These are separate because they need to be included, but putting
-# them as make dependencies would confuse make.
-VM_LINKS = image/bin image/lib image/sbin image/var/run
-
-VM_BUILD_FILES = build/etc/s6-rc
+BUILD_FILES = build/etc/s6-rc
build/empty:
mkdir -p $@
-build/rootfs.erofs: ../../../scripts/make-erofs.sh $(PACKAGES_FILE) $(VM_FILES) $(VM_BUILD_FILES) build/empty
+build/rootfs.erofs: ../../../scripts/make-erofs.sh $(PACKAGES_FILE) $(FILES) $(BUILD_FILES) build/empty file-list.mk
( \
cat $(PACKAGES_FILE) ;\
- for file in $(VM_FILES) $(VM_LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
- for file in $(VM_BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
- printf 'build/empty\n%s\n' $(VM_DIRS) ;\
+ for file in $(FILES) $(LINKS); do printf '%s\n%s\n' $$file "$${file#image/}"; done ;\
+ for file in $(BUILD_FILES); do printf '%s\n%s\n' $$file $${file#build/}; done ;\
+ printf 'build/empty\n%s\n' $(DIRS) ;\
) | ../../../scripts/make-erofs.sh $@
-VM_S6_RC_FILES = \
- image/etc/s6-rc/connman/dependencies.d/dbus \
- image/etc/s6-rc/connman/run \
- image/etc/s6-rc/connman/type \
- image/etc/s6-rc/dbus/notification-fd \
- image/etc/s6-rc/dbus/run \
- image/etc/s6-rc/dbus/type \
- image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
- image/etc/s6-rc/mdevd-coldplug/type \
- image/etc/s6-rc/mdevd-coldplug/up \
- image/etc/s6-rc/mdevd/notification-fd \
- image/etc/s6-rc/mdevd/run \
- image/etc/s6-rc/mdevd/type \
- image/etc/s6-rc/nftables/type \
- image/etc/s6-rc/nftables/up \
- image/etc/s6-rc/ok-all/contents \
- image/etc/s6-rc/ok-all/type \
- image/etc/s6-rc/sysctl/type \
- image/etc/s6-rc/sysctl/up
-
-build/etc/s6-rc: $(VM_S6_RC_FILES)
+build/etc/s6-rc: $(S6_RC_FILES) file-list.mk
mkdir -p $$(dirname $@)
rm -rf $@
diff --git a/vm/sys/net/file-list.mk b/vm/sys/net/file-list.mk
new file mode 100644
index 0000000000000000000000000000000000000000..de22e23c183f8d0a7df60a3c39395deae4f9a9bb
--- /dev/null
+++ b/vm/sys/net/file-list.mk
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: CC0-1.0
+# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com>
+# Generated by scripts/genfile.sh. Any changes will be overwritten.
+FILES ::= \
+ image/etc/dbus-1/system.conf \
+ image/etc/fstab \
+ image/etc/init \
+ image/etc/mdev.conf \
+ image/etc/mdev/iface \
+ image/etc/nftables.conf \
+ image/etc/passwd \
+ image/etc/s6-linux-init/run-image/service/getty-hvc0/run \
+ image/etc/s6-linux-init/scripts/rc.init \
+ image/etc/sysctl.conf
+
+# These are separate because they need to be included, but putting
+# them as make dependencies would confuse make.
+LINKS ::= \
+ image/bin \
+ image/lib \
+ image/sbin \
+ image/var/run
+
+S6_RC_FILES ::= \
+ image/etc/s6-rc/connman/dependencies.d/dbus \
+ image/etc/s6-rc/connman/run \
+ image/etc/s6-rc/connman/type \
+ image/etc/s6-rc/dbus/notification-fd \
+ image/etc/s6-rc/dbus/run \
+ image/etc/s6-rc/dbus/type \
+ image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \
+ image/etc/s6-rc/mdevd-coldplug/type \
+ image/etc/s6-rc/mdevd-coldplug/up \
+ image/etc/s6-rc/mdevd/notification-fd \
+ image/etc/s6-rc/mdevd/run \
+ image/etc/s6-rc/mdevd/type \
+ image/etc/s6-rc/nftables/type \
+ image/etc/s6-rc/nftables/up \
+ image/etc/s6-rc/ok-all/contents \
+ image/etc/s6-rc/ok-all/type \
+ image/etc/s6-rc/sysctl/type \
+ image/etc/s6-rc/sysctl/up
---
base-commit: 15ca6c4684313fcc9fcde3bda97d64698bb267ea
change-id: 20250831-genfiles-46787fc10869
--
Sincerely,
Demi Marie Obenour (she/her/hers)
next prev parent reply other threads:[~2025-09-21 3:01 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-09-04 1:56 [PATCH 0/4] Generate file lists from a script Demi Marie Obenour
2025-09-04 1:56 ` [PATCH 1/4] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-04 1:56 ` [PATCH 2/4] Generate makefile file lists from a script Demi Marie Obenour
2025-09-08 9:59 ` Alyssa Ross
2025-09-08 18:45 ` Demi Marie Obenour
2025-09-09 14:51 ` Alyssa Ross
2025-09-04 1:56 ` [PATCH 3/4] Common make rules for building erofs images Demi Marie Obenour
2025-09-08 10:01 ` Alyssa Ross
2025-09-08 18:53 ` Demi Marie Obenour
2025-09-09 14:56 ` Alyssa Ross
2025-09-04 1:56 ` [PATCH 4/4] Use /etc/s6-rc/compiled for compiled s6-rc directory Demi Marie Obenour
2025-09-10 5:29 ` [PATCH v2 0/3] Generate file lists from a script Demi Marie Obenour
2025-09-10 5:29 ` [PATCH v2 1/3] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-10 18:58 ` Alyssa Ross
2025-09-11 12:21 ` Demi Marie Obenour
2025-09-10 5:29 ` [PATCH v2 2/3] Generate makefile file lists from a script Demi Marie Obenour
2025-09-10 5:29 ` [PATCH v2 3/3] Common make rules for building erofs images Demi Marie Obenour
2025-09-11 12:47 ` [PATCH v3 0/4] Generate file lists from a script Demi Marie Obenour
2025-09-11 12:47 ` [PATCH v3 1/4] Do not ignore errors from tar Demi Marie Obenour
2025-09-17 11:48 ` Alyssa Ross
2025-09-18 2:45 ` Demi Marie Obenour
2025-09-19 7:46 ` Alyssa Ross
2025-09-30 12:59 ` Alyssa Ross
2025-09-19 7:55 ` Alyssa Ross
2025-09-19 19:03 ` Demi Marie Obenour
2025-09-11 12:47 ` [PATCH v3 2/4] Move all files for the image into a subdirectory Demi Marie Obenour
2025-09-17 12:30 ` Alyssa Ross
2025-09-17 12:39 ` Alyssa Ross
2025-09-17 13:03 ` Alyssa Ross
2025-09-11 12:47 ` [PATCH v3 3/4] Generate makefile file lists from a script Demi Marie Obenour
2025-09-11 12:47 ` [PATCH v3 4/4] Common make rules for building erofs images Demi Marie Obenour
2025-09-21 2:23 ` Demi Marie Obenour [this message]
2025-09-21 8:47 ` [PATCH v3] Generate file lists from a script Alyssa Ross
2025-09-21 16:51 ` Demi Marie Obenour
2025-09-21 17:07 ` Alyssa Ross
2025-09-21 17:24 ` [PATCH v4] " Demi Marie Obenour
2025-09-25 11:22 ` Alyssa Ross
2025-09-26 16:31 ` [PATCH v5] " Demi Marie Obenour
2025-09-27 8:19 ` Alyssa Ross
2025-09-27 8:42 ` Demi Marie Obenour
2025-09-27 16:22 ` [PATCH v6] " Demi Marie Obenour
2025-09-29 8:12 ` Alyssa Ross
2025-09-29 17:20 ` Demi Marie Obenour
2025-09-29 17:18 ` [PATCH v7] " Demi Marie Obenour
2025-10-01 9:20 ` Alyssa Ross
2025-10-01 9:24 ` Demi Marie Obenour
2025-10-01 9:35 ` Alyssa Ross
2025-10-01 18:30 ` [PATCH v8] " Demi Marie Obenour
2025-10-02 9:46 ` Alyssa Ross
2025-10-02 17:37 ` [PATCH v9] " Demi Marie Obenour
2025-10-03 9:04 ` Alyssa Ross
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250920-genfiles-v3-1-d6c2b6767b42@gmail.com \
--to=demiobenour@gmail.com \
--cc=devel@spectrum-os.org \
--cc=hi@alyssa.is \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://spectrum-os.org/git/crosvm
https://spectrum-os.org/git/doc
https://spectrum-os.org/git/mktuntap
https://spectrum-os.org/git/nixpkgs
https://spectrum-os.org/git/spectrum
https://spectrum-os.org/git/ucspi-vsock
https://spectrum-os.org/git/www
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).