[PATCH v3 0/4] Generate file lists from a script

Demi Marie Obenour
2025-09-11T12:51:13Z

Right now, the makefiles in host/rootfs, vm/sys/net, and img/app have
manually-maintained lists of files and symlinks. These duplicate the
information in the git repository and can easily get out of sync or
cause unnecessary merge conflicts.

Avoid this problem by moving the files used in the erofs images to an
image/ directory and using what Git knows about this directory to
generate the lists via a script. This makes maintenance easier and
avoids conflating code used in the build (anything not in image/) and
code for the image (anything that is in image/).

This reveals that there is a lot of duplicated make logic around image
generation, so consolidate it in a single makefile that is included by
all three of the makefiles.

Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
Changes in v3:
- EDITME: describe what is new in this series revision.
- EDITME: use bulletpoints and terse descriptions.
- Link to v2: https://lore.kernel.org/r/20250910-genfiles-v2-0-37ebe07a3cdc@gmail.com

Changes in v2:
- Drop the last patch (switching to /etc/s6-rc/compiled) as it is
controversial and should be reviewed separately.
- Add missing copyright notices.
- Use a wrapper shell script to make the awk code easier to read.
- Improve documentation.
- Add helper scripts for use in git hooks and rebasing.
- Link to v1: https://spectrum-os.org/lists/archives/spectrum-devel/20250903-genfiles-v1-0-cc993fcb1e4c@gmail.com/

---
Demi Marie Obenour (4):
Do not ignore errors from tar
Move all files for the image into a subdirectory
Generate makefile file lists from a script
Common make rules for building erofs images

Documentation/development/built-in-vms.adoc | 17 +++
host/rootfs/Makefile | 141 +--------------------
host/rootfs/default.nix | 1 +
host/rootfs/file-list.mk | 100 +++++++++++++++
host/rootfs/{ => image}/bin | 0
host/rootfs/{ => image}/etc/fonts/fonts.conf | 0
host/rootfs/{ => image}/etc/fstab | 0
host/rootfs/{ => image}/etc/group | 0
host/rootfs/{ => image}/etc/group.license | 0
host/rootfs/{ => image}/etc/init | 0
host/rootfs/{ => image}/etc/login | 0
host/rootfs/{ => image}/etc/mdev.conf | 0
host/rootfs/{ => image}/etc/mdev/listen | 0
host/rootfs/{ => image}/etc/mdev/net/add | 0
host/rootfs/{ => image}/etc/mdev/wait | 0
host/rootfs/{ => image}/etc/parse-devname | 0
host/rootfs/{ => image}/etc/passwd | 0
host/rootfs/{ => image}/etc/passwd.license | 0
.../etc/s6-linux-init/env/WAYLAND_DISPLAY | 0
.../etc/s6-linux-init/env/WAYLAND_DISPLAY.license | 0
.../etc/s6-linux-init/env/XDG_RUNTIME_DIR | 0
.../etc/s6-linux-init/env/XDG_RUNTIME_DIR.license | 0
.../etc/s6-linux-init/run-image/opengl-driver | 0
.../run-image/service/dbus/notification-fd | 0
.../run-image/service/dbus/notification-fd.license | 0
.../etc/s6-linux-init/run-image/service/dbus/run | 0
.../service/dbus/template/notification-fd | 0
.../service/dbus/template/notification-fd.license | 0
.../run-image/service/dbus/template/run | 0
.../s6-linux-init/run-image/service/getty-tty1/run | 0
.../s6-linux-init/run-image/service/getty-tty2/run | 0
.../s6-linux-init/run-image/service/getty-tty3/run | 0
.../s6-linux-init/run-image/service/getty-tty4/run | 0
.../service/s6-svscan-log/notification-fd | 0
.../service/s6-svscan-log/notification-fd.license | 0
.../run-image/service/s6-svscan-log/run | 0
.../run-image/service/serial-getty-generator/run | 0
.../run-image/service/serial-getty/notification-fd | 0
.../service/serial-getty/notification-fd.license | 0
.../run-image/service/serial-getty/run | 0
.../run-image/service/serial-getty/template/run | 0
.../service/vhost-user-fs/notification-fd | 0
.../service/vhost-user-fs/notification-fd.license | 0
.../run-image/service/vhost-user-fs/run | 0
.../service/vhost-user-fs/template/notification-fd | 0
.../vhost-user-fs/template/notification-fd.license | 0
.../run-image/service/vhost-user-fs/template/run | 0
.../service/vhost-user-gpu/notification-fd | 0
.../service/vhost-user-gpu/notification-fd.license | 0
.../run-image/service/vhost-user-gpu/run | 0
.../service/vhost-user-gpu/template/data/check | 0
.../vhost-user-gpu/template/notification-fd | 0
.../template/notification-fd.license | 0
.../run-image/service/vhost-user-gpu/template/run | 0
.../run-image/service/vhost-user-gpu/template/type | 0
.../service/vhost-user-gpu/template/type.license | 0
.../run-image/service/vmm/notification-fd | 0
.../run-image/service/vmm/notification-fd.license | 0
.../etc/s6-linux-init/run-image/service/vmm/run | 0
.../run-image/service/vmm/template/notification-fd | 0
.../service/vmm/template/notification-fd.license | 0
.../run-image/service/vmm/template/run | 0
.../notification-fd | 0
.../notification-fd.license | 0
.../service/xdg-desktop-portal-spectrum-host/run | 0
.../template/notification-fd | 0
.../template/notification-fd.license | 0
.../xdg-desktop-portal-spectrum-host/template/run | 0
.../{ => image}/etc/s6-linux-init/scripts/rc.init | 0
host/rootfs/{ => image}/etc/s6-rc/card0/type | 0
.../{ => image}/etc/s6-rc/card0/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/card0/up | 0
host/rootfs/{ => image}/etc/s6-rc/core/type | 0
.../rootfs/{ => image}/etc/s6-rc/core/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/core/up | 0
host/rootfs/{ => image}/etc/s6-rc/kvm/timeout-up | 0
.../{ => image}/etc/s6-rc/kvm/timeout-up.license | 0
host/rootfs/{ => image}/etc/s6-rc/kvm/type | 0
host/rootfs/{ => image}/etc/s6-rc/kvm/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/kvm/up | 0
.../etc/s6-rc/mdevd-coldplug/dependencies | 0
.../{ => image}/etc/s6-rc/mdevd-coldplug/type | 0
.../etc/s6-rc/mdevd-coldplug/type.license | 0
.../rootfs/{ => image}/etc/s6-rc/mdevd-coldplug/up | 0
.../{ => image}/etc/s6-rc/mdevd/notification-fd | 0
.../etc/s6-rc/mdevd/notification-fd.license | 0
host/rootfs/{ => image}/etc/s6-rc/mdevd/run | 0
host/rootfs/{ => image}/etc/s6-rc/mdevd/type | 0
.../{ => image}/etc/s6-rc/mdevd/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/ok-all/contents | 0
host/rootfs/{ => image}/etc/s6-rc/ok-all/type | 0
.../{ => image}/etc/s6-rc/ok-all/type.license | 0
.../rootfs/{ => image}/etc/s6-rc/static-nodes/type | 0
.../etc/s6-rc/static-nodes/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/static-nodes/up | 0
.../{ => image}/etc/s6-rc/sys-vmms/dependencies | 0
host/rootfs/{ => image}/etc/s6-rc/sys-vmms/type | 0
.../{ => image}/etc/s6-rc/sys-vmms/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/sys-vmms/up | 0
host/rootfs/{ => image}/etc/s6-rc/vm-env/contents | 0
host/rootfs/{ => image}/etc/s6-rc/vm-env/type | 0
.../{ => image}/etc/s6-rc/vm-env/type.license | 0
host/rootfs/{ => image}/etc/s6-rc/vmm-env/contents | 0
host/rootfs/{ => image}/etc/s6-rc/vmm-env/type | 0
.../{ => image}/etc/s6-rc/vmm-env/type.license | 0
.../{ => image}/etc/s6-rc/weston/dependencies | 0
.../{ => image}/etc/s6-rc/weston/notification-fd | 0
.../etc/s6-rc/weston/notification-fd.license | 0
host/rootfs/{ => image}/etc/s6-rc/weston/run | 0
host/rootfs/{ => image}/etc/s6-rc/weston/type | 0
.../{ => image}/etc/s6-rc/weston/type.license | 0
host/rootfs/{ => image}/etc/xdg/weston/autolaunch | 0
host/rootfs/{ => image}/etc/xdg/weston/weston.ini | 0
host/rootfs/{ => image}/lib | 0
host/rootfs/{ => image}/sbin | 0
host/rootfs/{ => image}/usr/bin/assign-devices | 0
.../{ => image}/usr/bin/create-vm-dependencies | 0
host/rootfs/{ => image}/usr/bin/run-appimage | 0
host/rootfs/{ => image}/usr/bin/run-vmm | 0
host/rootfs/{ => image}/usr/bin/vm-console | 0
host/rootfs/{ => image}/usr/bin/vm-import | 0
host/rootfs/{ => image}/usr/bin/vm-start | 0
host/rootfs/{ => image}/usr/bin/vm-stop | 0
host/rootfs/{ => image}/usr/bin/xdg-open | 0
.../org.freedesktop.portal.Documents.service | 0
img/app/Makefile | 98 +-------------
img/app/default.nix | 1 +
img/app/file-list.mk | 65 ++++++++++
img/app/{ => image}/bin | 0
img/app/{ => image}/etc/dbus-1/session.conf | 0
img/app/{ => image}/etc/fonts/fonts.conf | 0
img/app/{ => image}/etc/fstab | 0
img/app/{ => image}/etc/init | 0
img/app/{ => image}/etc/mdev.conf | 0
img/app/{ => image}/etc/mdev/iface | 0
img/app/{ => image}/etc/mdev/listen | 0
img/app/{ => image}/etc/mdev/virtiofs | 0
img/app/{ => image}/etc/mdev/wait | 0
img/app/{ => image}/etc/passwd | 0
img/app/{ => image}/etc/passwd.license | 0
img/app/{ => image}/etc/pipewire/pipewire.conf | 0
img/app/{ => image}/etc/resolv.conf | 0
.../etc/s6-linux-init/env/DBUS_SESSION_BUS_ADDRESS | 0
.../env/DBUS_SESSION_BUS_ADDRESS.license | 0
img/app/{ => image}/etc/s6-linux-init/env/DISPLAY | 0
.../etc/s6-linux-init/env/DISPLAY.license | 0
.../etc/s6-linux-init/env/GTK_USE_PORTAL | 0
.../etc/s6-linux-init/env/GTK_USE_PORTAL.license | 0
.../s6-linux-init/env/NIX_XDG_DESKTOP_PORTAL_DIR | 0
.../env/NIX_XDG_DESKTOP_PORTAL_DIR.license | 0
.../etc/s6-linux-init/env/WAYLAND_DISPLAY | 0
.../etc/s6-linux-init/env/WAYLAND_DISPLAY.license | 0
.../etc/s6-linux-init/env/XDG_RUNTIME_DIR | 0
.../etc/s6-linux-init/env/XDG_RUNTIME_DIR.license | 0
.../s6-linux-init/run-image/service/getty-hvc0/run | 0
.../s6-linux-init-shutdownd/notification-fd | 0
.../notification-fd.license | 0
.../run-image/service/s6-linux-init-shutdownd/run | 0
.../{ => image}/etc/s6-linux-init/scripts/rc.init | 0
.../etc/s6-linux-init/scripts/rc.shutdown | 0
.../etc/s6-linux-init/scripts/rc.shutdown.final | 0
.../{ => image}/etc/s6-rc/app/dependencies.d/dbus | 0
.../etc/s6-rc/app/dependencies.d/pipewire | 0
.../s6-rc/app/dependencies.d/wayland-proxy-virtwl | 0
img/app/{ => image}/etc/s6-rc/app/run | 0
img/app/{ => image}/etc/s6-rc/app/type | 0
img/app/{ => image}/etc/s6-rc/app/type.license | 0
img/app/{ => image}/etc/s6-rc/dbus/notification-fd | 0
.../etc/s6-rc/dbus/notification-fd.license | 0
img/app/{ => image}/etc/s6-rc/dbus/run | 0
img/app/{ => image}/etc/s6-rc/dbus/type | 0
img/app/{ => image}/etc/s6-rc/dbus/type.license | 0
.../etc/s6-rc/mdevd-coldplug/dependencies | 0
img/app/{ => image}/etc/s6-rc/mdevd-coldplug/type | 0
.../etc/s6-rc/mdevd-coldplug/type.license | 0
img/app/{ => image}/etc/s6-rc/mdevd-coldplug/up | 0
.../{ => image}/etc/s6-rc/mdevd/notification-fd | 0
.../etc/s6-rc/mdevd/notification-fd.license | 0
img/app/{ => image}/etc/s6-rc/mdevd/run | 0
img/app/{ => image}/etc/s6-rc/mdevd/type | 0
img/app/{ => image}/etc/s6-rc/mdevd/type.license | 0
img/app/{ => image}/etc/s6-rc/ok-all/contents | 0
img/app/{ => image}/etc/s6-rc/ok-all/type | 0
img/app/{ => image}/etc/s6-rc/ok-all/type.license | 0
.../{ => image}/etc/s6-rc/pipewire/notification-fd | 0
.../etc/s6-rc/pipewire/notification-fd.license | 0
img/app/{ => image}/etc/s6-rc/pipewire/run | 0
img/app/{ => image}/etc/s6-rc/pipewire/type | 0
.../{ => image}/etc/s6-rc/pipewire/type.license | 0
.../etc/s6-rc/wayland-proxy-virtwl/notification-fd | 0
.../wayland-proxy-virtwl/notification-fd.license | 0
.../{ => image}/etc/s6-rc/wayland-proxy-virtwl/run | 0
.../etc/s6-rc/wayland-proxy-virtwl/type | 0
.../etc/s6-rc/wayland-proxy-virtwl/type.license | 0
.../etc/s6-rc/wireplumber/dependencies.d/dbus | 0
.../etc/s6-rc/wireplumber/dependencies.d/pipewire | 0
img/app/{ => image}/etc/s6-rc/wireplumber/run | 0
img/app/{ => image}/etc/s6-rc/wireplumber/type | 0
.../{ => image}/etc/s6-rc/wireplumber/type.license | 0
.../{ => image}/etc/ssl/certs/ca-certificates.crt | 0
.../wireplumber.conf.d/99_spectrum.conf | 0
.../etc/xdg/xdg-desktop-portal/portals.conf | 0
img/app/{ => image}/sbin | 0
lib/common.mk | 1 +
lib/erofs.mk | 49 +++++++
scripts/genfiles.awk | 120 ++++++++++++++++++
scripts/genfiles.sh | 23 ++++
scripts/git-rebase | 17 +++
scripts/pre-commit.sh | 9 ++
vm/sys/net/Makefile | 69 +---------
vm/sys/net/default.nix | 1 +
vm/sys/net/file-list.mk | 42 ++++++
vm/sys/net/{ => image}/bin | 0
vm/sys/net/{ => image}/etc/dbus-1/system.conf | 0
vm/sys/net/{ => image}/etc/fstab | 0
vm/sys/net/{ => image}/etc/init | 0
vm/sys/net/{ => image}/etc/mdev.conf | 0
vm/sys/net/{ => image}/etc/mdev/iface | 0
vm/sys/net/{ => image}/etc/nftables.conf | 0
vm/sys/net/{ => image}/etc/passwd | 0
vm/sys/net/{ => image}/etc/passwd.license | 0
.../s6-linux-init/run-image/service/getty-hvc0/run | 0
.../{ => image}/etc/s6-linux-init/scripts/rc.init | 0
.../net/{ => image}/etc/s6-rc/connman/dependencies | 0
vm/sys/net/{ => image}/etc/s6-rc/connman/run | 0
vm/sys/net/{ => image}/etc/s6-rc/connman/type | 0
.../net/{ => image}/etc/s6-rc/connman/type.license | 0
.../net/{ => image}/etc/s6-rc/dbus/notification-fd | 0
.../etc/s6-rc/dbus/notification-fd.license | 0
vm/sys/net/{ => image}/etc/s6-rc/dbus/run | 0
vm/sys/net/{ => image}/etc/s6-rc/dbus/type | 0
vm/sys/net/{ => image}/etc/s6-rc/dbus/type.license | 0
.../etc/s6-rc/mdevd-coldplug/dependencies | 0
.../net/{ => image}/etc/s6-rc/mdevd-coldplug/type | 0
.../etc/s6-rc/mdevd-coldplug/type.license | 0
vm/sys/net/{ => image}/etc/s6-rc/mdevd-coldplug/up | 0
.../{ => image}/etc/s6-rc/mdevd/notification-fd | 0
.../etc/s6-rc/mdevd/notification-fd.license | 0
vm/sys/net/{ => image}/etc/s6-rc/mdevd/run | 0
vm/sys/net/{ => image}/etc/s6-rc/mdevd/type | 0
.../net/{ => image}/etc/s6-rc/mdevd/type.license | 0
vm/sys/net/{ => image}/etc/s6-rc/nftables/type | 0
.../{ => image}/etc/s6-rc/nftables/type.license | 0
vm/sys/net/{ => image}/etc/s6-rc/nftables/up | 0
vm/sys/net/{ => image}/etc/s6-rc/ok-all/contents | 0
vm/sys/net/{ => image}/etc/s6-rc/ok-all/type | 0
.../net/{ => image}/etc/s6-rc/ok-all/type.license | 0
vm/sys/net/{ => image}/etc/s6-rc/sysctl/type | 0
.../net/{ => image}/etc/s6-rc/sysctl/type.license | 0
vm/sys/net/{ => image}/etc/s6-rc/sysctl/up | 0
vm/sys/net/{ => image}/etc/sysctl.conf | 0
vm/sys/net/{ => image}/lib | 0
vm/sys/net/{ => image}/sbin | 0
vm/sys/net/{ => image}/var/run | 0
254 files changed, 457 insertions(+), 297 deletions(-)
---
base-commit: cbb10b15f36a7ac35400e226b284b1cb091cb3c9
change-id: 20250831-genfiles-46787fc10869
--
Sincerely,
Demi Marie Obenour (she/her/hers)