[PATCH] host/rootfs: Sandbox Cloud Hypervisor

[PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Demi Marie Obenour]

It only needs access to a small number of resources.  Unfortunately, it
needs access to /dev/vfio right now.  This should be fixed by using file
descriptor passing instead.  Also, Cloud Hypervisor should not run as
root.

Cloud Hypervisor needs to be able to lock memory.  Running in a user
namespace prevents it from using CAP_IPC_LOCK.  Therefore, it is
necessary to increase RLIMIT_MLOCK before running Cloud Hypervisor.

Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
 host/rootfs/image/usr/bin/run-vmm | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm
index ba8b59c2677408acdd01c2eda3cf2dd60992d881..24c3d607bfcf6fea6196b61d2941141486d33fd6 100755
--- a/host/rootfs/image/usr/bin/run-vmm
+++ b/host/rootfs/image/usr/bin/run-vmm
@@ -52,5 +52,36 @@ unexport !
 fdmove -c 3 0
 redirfd -r 0 /dev/null
 
+s6-softlimit -H -l 18446744073709551615
 if { udevadm wait /dev/kvm }
-cloud-hypervisor --api-socket fd=3
+bwrap
+  --unshare-all
+  --unshare-user
+  --dev /dev
+  --dev-bind /dev/kvm /dev/kvm
+  --dev-bind /dev/vfio /dev/vfio
+  --tmpfs /dev/shm
+  --tmpfs /tmp
+  --tmpfs /var/tmp
+  --ro-bind /etc /etc
+  --ro-bind /lib /lib
+  --ro-bind /nix /nix
+  --ro-bind /usr /usr
+  --ro-bind /sys /sys
+  --bind /run /run
+  --proc /proc
+  --ro-bind /proc/sys /proc/sys
+  --tmpfs /proc/scsi
+  --remount-ro /proc/scsi
+  --tmpfs /proc/acpi
+  --remount-ro /proc/acpi
+  --tmpfs /proc/fs
+  --remount-ro /proc/fs
+  --tmpfs /proc/irq
+  --remount-ro /proc/irq
+  --ro-bind /dev/null /proc/timer_list
+  --ro-bind /dev/null /proc/kcore
+  --ro-bind /dev/null /proc/kallsyms
+  --ro-bind /dev/null /proc/sysrq-trigger
+  --
+  cloud-hypervisor --api-socket fd=3

---
base-commit: 92e219e7c08c479d216a46d2736ea9d229ff034d
change-id: 20251206-b4-sandbox-9be7e5ed9926

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

Re: [PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> It only needs access to a small number of resources.  Unfortunately, it
> needs access to /dev/vfio right now.  This should be fixed by using file
> descriptor passing instead.  Also, Cloud Hypervisor should not run as
> root.
>
> Cloud Hypervisor needs to be able to lock memory.  Running in a user
> namespace prevents it from using CAP_IPC_LOCK.  Therefore, it is
> necessary to increase RLIMIT_MLOCK before running Cloud Hypervisor.
>
> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
> ---
>  host/rootfs/image/usr/bin/run-vmm | 33 ++++++++++++++++++++++++++++++++-
>  1 file changed, 32 insertions(+), 1 deletion(-)
>
> diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm
> index ba8b59c2677408acdd01c2eda3cf2dd60992d881..24c3d607bfcf6fea6196b61d2941141486d33fd6 100755
> --- a/host/rootfs/image/usr/bin/run-vmm
> +++ b/host/rootfs/image/usr/bin/run-vmm
> @@ -52,5 +52,36 @@ unexport !
>  fdmove -c 3 0
>  redirfd -r 0 /dev/null
>  
> +s6-softlimit -H -l 18446744073709551615

My question about the limit from last time is still waiting for an
answer…

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 1404 bytes --]

On 12/6/25 12:29, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> It only needs access to a small number of resources.  Unfortunately, it
>> needs access to /dev/vfio right now.  This should be fixed by using file
>> descriptor passing instead.  Also, Cloud Hypervisor should not run as
>> root.
>>
>> Cloud Hypervisor needs to be able to lock memory.  Running in a user
>> namespace prevents it from using CAP_IPC_LOCK.  Therefore, it is
>> necessary to increase RLIMIT_MLOCK before running Cloud Hypervisor.
>>
>> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
>> ---
>>  host/rootfs/image/usr/bin/run-vmm | 33 ++++++++++++++++++++++++++++++++-
>>  1 file changed, 32 insertions(+), 1 deletion(-)
>>
>> diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm
>> index ba8b59c2677408acdd01c2eda3cf2dd60992d881..24c3d607bfcf6fea6196b61d2941141486d33fd6 100755
>> --- a/host/rootfs/image/usr/bin/run-vmm
>> +++ b/host/rootfs/image/usr/bin/run-vmm
>> @@ -52,5 +52,36 @@ unexport !
>>  fdmove -c 3 0
>>  redirfd -r 0 /dev/null
>>  
>> +s6-softlimit -H -l 18446744073709551615
> 
> My question about the limit from last time is still waiting for an
> answer…

Whoops, I saw that all the other patches had been applied and missed
that this one hadn't been.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2775 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> It only needs access to a small number of resources.  Unfortunately, it
> needs access to /dev/vfio right now.  This should be fixed by using file
> descriptor passing instead.  Also, Cloud Hypervisor should not run as
> root.
>
> Cloud Hypervisor needs to be able to lock memory.  Running in a user
> namespace prevents it from using CAP_IPC_LOCK.  Therefore, it is
> necessary to increase RLIMIT_MLOCK before running Cloud Hypervisor.
>
> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
> ---
>  host/rootfs/image/usr/bin/run-vmm | 33 ++++++++++++++++++++++++++++++++-
>  1 file changed, 32 insertions(+), 1 deletion(-)

Looks good, but it seems to only work for VMs run as s6 services, not
those run through run-appimage or run-flatpak.  (I suppose the appimage
integration test is somehow not thorough enough to catch this, which
should be easier to fix once we understand the problem.)

bwrap: execvp cloud-hypervisor: No such file or directory

I suppose this is because etc/login sets PATH to /bin, and we don't
share /bin with the sandbox.  Changing that to /usr/bin would be a good
idea I suppose, but would it also be a good idea to share /bin with the
sandbox?  What do you think?

> diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm
> index ba8b59c2677408acdd01c2eda3cf2dd60992d881..24c3d607bfcf6fea6196b61d2941141486d33fd6 100755
> --- a/host/rootfs/image/usr/bin/run-vmm
> +++ b/host/rootfs/image/usr/bin/run-vmm
> @@ -52,5 +52,36 @@ unexport !
>  fdmove -c 3 0
>  redirfd -r 0 /dev/null
>  
> +s6-softlimit -H -l 18446744073709551615
>  if { udevadm wait /dev/kvm }
> -cloud-hypervisor --api-socket fd=3
> +bwrap
> +  --unshare-all
> +  --unshare-user
> +  --dev /dev
> +  --dev-bind /dev/kvm /dev/kvm
> +  --dev-bind /dev/vfio /dev/vfio
> +  --tmpfs /dev/shm
> +  --tmpfs /tmp
> +  --tmpfs /var/tmp
> +  --ro-bind /etc /etc
> +  --ro-bind /lib /lib
> +  --ro-bind /nix /nix
> +  --ro-bind /usr /usr
> +  --ro-bind /sys /sys
> +  --bind /run /run
> +  --proc /proc
> +  --ro-bind /proc/sys /proc/sys
> +  --tmpfs /proc/scsi
> +  --remount-ro /proc/scsi
> +  --tmpfs /proc/acpi
> +  --remount-ro /proc/acpi
> +  --tmpfs /proc/fs
> +  --remount-ro /proc/fs
> +  --tmpfs /proc/irq
> +  --remount-ro /proc/irq
> +  --ro-bind /dev/null /proc/timer_list
> +  --ro-bind /dev/null /proc/kcore
> +  --ro-bind /dev/null /proc/kallsyms
> +  --ro-bind /dev/null /proc/sysrq-trigger
> +  --
> +  cloud-hypervisor --api-socket fd=3
>
> ---
> base-commit: 92e219e7c08c479d216a46d2736ea9d229ff034d
> change-id: 20251206-b4-sandbox-9be7e5ed9926
>
> -- 
> Sincerely,
> Demi Marie Obenour (she/her/hers)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 2991 bytes --]

On 12/9/25 04:35, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> It only needs access to a small number of resources.  Unfortunately, it
>> needs access to /dev/vfio right now.  This should be fixed by using file
>> descriptor passing instead.  Also, Cloud Hypervisor should not run as
>> root.
>>
>> Cloud Hypervisor needs to be able to lock memory.  Running in a user
>> namespace prevents it from using CAP_IPC_LOCK.  Therefore, it is
>> necessary to increase RLIMIT_MLOCK before running Cloud Hypervisor.
>>
>> Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
>> ---
>>  host/rootfs/image/usr/bin/run-vmm | 33 ++++++++++++++++++++++++++++++++-
>>  1 file changed, 32 insertions(+), 1 deletion(-)
> 
> Looks good, but it seems to only work for VMs run as s6 services, not
> those run through run-appimage or run-flatpak.  (I suppose the appimage
> integration test is somehow not thorough enough to catch this, which
> should be easier to fix once we understand the problem.)
> 
> bwrap: execvp cloud-hypervisor: No such file or directory
> 
> I suppose this is because etc/login sets PATH to /bin, and we don't
> share /bin with the sandbox.  Changing that to /usr/bin would be a good
> idea I suppose, but would it also be a good idea to share /bin with the
> sandbox?  What do you think?

+1 on sharing /bin and /sbin.

>> diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm
>> index ba8b59c2677408acdd01c2eda3cf2dd60992d881..24c3d607bfcf6fea6196b61d2941141486d33fd6 100755
>> --- a/host/rootfs/image/usr/bin/run-vmm
>> +++ b/host/rootfs/image/usr/bin/run-vmm
>> @@ -52,5 +52,36 @@ unexport !
>>  fdmove -c 3 0
>>  redirfd -r 0 /dev/null
>>  
>> +s6-softlimit -H -l 18446744073709551615
>>  if { udevadm wait /dev/kvm }
>> -cloud-hypervisor --api-socket fd=3
>> +bwrap
>> +  --unshare-all
>> +  --unshare-user
>> +  --dev /dev
>> +  --dev-bind /dev/kvm /dev/kvm
>> +  --dev-bind /dev/vfio /dev/vfio
>> +  --tmpfs /dev/shm
>> +  --tmpfs /tmp
>> +  --tmpfs /var/tmp
>> +  --ro-bind /etc /etc
>> +  --ro-bind /lib /lib
>> +  --ro-bind /nix /nix
>> +  --ro-bind /usr /usr
>> +  --ro-bind /sys /sys
>> +  --bind /run /run
>> +  --proc /proc
>> +  --ro-bind /proc/sys /proc/sys
>> +  --tmpfs /proc/scsi
>> +  --remount-ro /proc/scsi
>> +  --tmpfs /proc/acpi
>> +  --remount-ro /proc/acpi
>> +  --tmpfs /proc/fs
>> +  --remount-ro /proc/fs
>> +  --tmpfs /proc/irq
>> +  --remount-ro /proc/irq
>> +  --ro-bind /dev/null /proc/timer_list
>> +  --ro-bind /dev/null /proc/kcore
>> +  --ro-bind /dev/null /proc/kallsyms
>> +  --ro-bind /dev/null /proc/sysrq-trigger
>> +  --
>> +  cloud-hypervisor --api-socket fd=3
>>
>> ---
>> base-commit: 92e219e7c08c479d216a46d2736ea9d229ff034d
>> change-id: 20251206-b4-sandbox-9be7e5ed9926
>>
>> -- 
>> Sincerely,
>> Demi Marie Obenour (she/her/hers)


-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] host/rootfs: Sandbox Cloud Hypervisor

[Alyssa Ross]

This patch has been committed as ec47d362e3053d30668976f00bfd31b539cfd9a7,
which can be viewed online at
https://spectrum-os.org/git/spectrum/commit/?id=ec47d362e3053d30668976f00bfd31b539cfd9a7.

This is an automated message.  Send comments/questions/requests to:
Alyssa Ross <hi@alyssa.is>