Re: Networking test runs run-vmm twice with the same VM ID

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 3196 bytes --]

On 7/16/26 04:29, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> The networking test runs run-vmm twice with the same VM ID.
>> If it waits for the previous cgroup to be empty, the second
>> call waits forever.
>>
>> Is this expected?  I would expect that run-vmm is only called
>> once.
> 
> Assuming you mean run twice in sequence, rather than run twice at the
> same time, then yes, it is expected.
> 
> The way it works at the moment is that imported VMs have a VM ID
> assigned when they're imported, and they can then be started any number
> of times.  When the VM shuts down, the VMM exits, so if the VM is
> started again after that, run-vmm will be run again with the same VM ID.
> If this were not the case, VM IDs could not be assigned at import time,
> and stay the same between successive boots of the same VM.
> 
> If your cgroup is not being emptied as expected, that is a bug,
> regardless of whether it ends up being used again, because there
> shouldn't be lingering processes or cgroups hanging around forever.
> It sounds like we should have a test that confirms that this doesn't
> happen, if it's hard to get right.
> 
> The networking test does do something slightly unusual, which is that it
> restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
> Hypervisor API.  I would expect that this runs the finish script, but
> might be worth confirming.

It does indeed run the finish script!

It appears there are likely two or more bugs.  The first is that
sending a signal to the run-vmm process itself is insufficient:
it leaves child processes behind.  I don't know why that is.  Is it
something you could figure out?

The second bug was that the finish script purged the wrong cgroup.
It tried to purge /vmm, which doesn't exist.  Purging a nonexistent
cgroup is a (intentionally) a no-op, so the finish script failed to
do anything.

Because of these two problems, there were still processes left in the
cgroup when run-vmm ran again.  Therefore, cgroup-setup was waiting
for them to exit, which never happened.

I haven't fixed the first problem, but I did fix the second.  However,
this caused /run/vsock/${1}/vsock to be left behind.  I suspect this
is because cgroup-setup sends SIGKILL, so Cloud Hypervisor never gets
to unlink the socket.  This caused Cloud Hypervisor to fail to restart.

To solve *that* problem, I had to add `rm -f /run/vsock/${1}/vsock`
to run-vmm.  Fortunately, this is safe against symlink attacks: the
VMM has no write access to /run/vsock/${1}, so it can't turn that
into a symlink, and rm doesn't follow symlinks in the file it is told
to delete.

Once I added the rm call, the tests passed!

This means that the Rust code was actually working fine, and that the
way it waits for processes to exit works properly.  The problem was
in the surrounding execline scripting.  While the Rust code *was*
missing locking, that was not the cause of any of the problems in
the test suite.

Do you have suggestions for fixing run-vmm and/or preventing similar
problems in the future?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]