Demi Marie Obenour <demiobenour@gmail.com> writes:

> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
> ---
>  Documentation/doc/development/control-groups.adoc | 88 +++++++++++++++++++++++
>  1 file changed, 88 insertions(+)

CCing Valentin for documentation oversight.  I found this very clearly
written and easy to read. :)

> diff --git a/Documentation/doc/development/control-groups.adoc b/Documentation/doc/development/control-groups.adoc
> new file mode 100644
> index 0000000000000000000000000000000000000000..6ce33f21a230d012a690fc5deb2ba597d1d01ef1
> --- /dev/null
> +++ b/Documentation/doc/development/control-groups.adoc
> @@ -0,0 +1,88 @@
> += Control groups in Spectrum
> +
> +// SPDX-FileCopyrightText: 2026 Demi Marie Obenour <demiobenour@gmail.com>
> +// SPDX-License-Identifier: GFDL-1.3-no-invariants-or-later OR CC-BY-SA-4.0
> +
> +Linux control groups (cgroups) can be used for several purposes:
> +
> +1. They allow waiting for a group of processes to exit.
> +2. They allow terminating a group of processes.
> +3. They allow limiting a group of processes' access to resources.
> +
> +Spectrum currently uses the first two.  The third is not yet used,
> +but will be in the future.
> +
> +== Control Group Hierarchy
> +
> +Spectrum uses the following cgroup hierarchy:
> +
> +1. There is a `/vm-services.slice` cgroup that contains all the per-VM
> +   services on the system.
> +2. The per-VM services for each VM are under `/vm-services.slice/vm-${VM}.slice`,
> +   where `${VM}` is replaced by the VM's ID.
> +3. Each per-VM service is under `/vm-services.slice/vm-${VM}.slice/${SERVICE_NAME}`,
> +   where `${VM}` is replaced by the VM's ID and `${SERVICE_NAME}` is replaced by
> +   the name of the service.
> +4. The VMM runs under `/vm-services.slice/vm-${VM}.slice/vmm`.
> +
> +If a cgroup contains child cgroups, it likely contains a `$inner.service`
> +cgroup.  This is where programs that would otherwise run in the cgroup itself
> +are placed.  Generally, these programs are instances of `s6-svscan` and/or
> +`s6-supervise`.

This section doesn't mention where non-per-VM services go.  It might
also be nice to explicitly mention the "no internal processes" rule as
the reason for $inner.service, in case people are unfamiliar with it.

> +== Using Control Groups
> +
> +When adding a new s6 service, one should carefully consider whether it
> +should be placed in a control group.  Most services should be placed in
> +a control group, with only a few exceptions:
> +
> +1. Services, such as `getty`, that spawn background processes.
> +2. Loggers.
> +3. Trivial services that don't do anything.

We shouldn't have any services that don't do anything!  Maybe we could
be more specific?  Or just say "trivial services where cgroup are
unnecessary" or something.

> +
> +Generally, it's best to set the control group up as the first thing
> +the service does.  To do that, use `cgroup-setup --leaf -- $1 COMMAND_LINE`,

The " -- " here gets turned into an en-dash when rendered, so must need
to be escaped somehow.

> +where `$1` should be the service name and `COMMAND_LINE` is the program
> +to run in a cgroup.
> +
> +If you use execline for your run script, this is as simple as:
> +
> +.run
> +....
> +#!/bin/execlineb -WS1
> +
> +cgroup-setup --leaf -- $1
> +# rest of script comes here
> +....
> +
> +If the service exits, it's usually best to terminate any programs left
> +behind with SIGKILL and remove the control group.  In Spectrum, this is
> +called "purging" the cgroup.  To purge the cgroup when a service exits,
> +make the `finish` script invoke `/usr/bin/cgroup-s6-finish`.  The first
> +two command line arguments must be the first two arguments passed to the
> +`finish` script.  The third argument must be the path to the cgroup to
> +be purged relative to the cgroup the program itself is in.  This is
> +usually, but not always, the third argument to the `finish` script.
> +
> +When invoked as `cgroup-s6-finish`, `cgroup-setup` checks if
> +the service exited due to a signal that caused it to dump core.  If it
> +did, `cgroup-s6-finish` exits with status 125, ensuring that
> +`s6-supervise` will *not* restart it.  This is intentional: if a service
> +crashes due to a fatal signal, this is possibly a sign of memory
> +corruption.  Restarting the service in this case can turn an unreliable
> +memory corruption exploit into a reliable one.  Rust panics do not cause
> +core dumps, so the service will be restarted afterwards.

(Just noting that in the review of cgroup-setup itself, I said that to
me this does not seem likely something that should be part of
cgroup-setup itself.  If that does change, this will need to be updated.)

> +One can also use `cgroup-purge` to purge a cgroup explicitly.  This is
> +used to stop the VMM and all per-VM services when a VM is shut down.
> +
> +== Future plans
> +
> +Control groups are designed around a single writer process controlling each
> +of them.  Many Linux distros use systemd for this, but Spectrum doesn't use
> +systemd.  The only persistent per-service process is s6-supervise, but that
> +doesn't have control group support.
> +
> +Instead, the plan is to have a database containing this information.
> +Whether this will be in the `data/` subdirectory of the service directory
> +or a separate system-wide database has not yet been determined.