Networking test runs run-vmm twice with the same VM ID

Networking test runs run-vmm twice with the same VM ID

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 269 bytes --]

The networking test runs run-vmm twice with the same VM ID.
If it waits for the previous cgroup to be empty, the second
call waits forever.

Is this expected?  I would expect that run-vmm is only called
once.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> The networking test runs run-vmm twice with the same VM ID.
> If it waits for the previous cgroup to be empty, the second
> call waits forever.
>
> Is this expected?  I would expect that run-vmm is only called
> once.

Assuming you mean run twice in sequence, rather than run twice at the
same time, then yes, it is expected.

The way it works at the moment is that imported VMs have a VM ID
assigned when they're imported, and they can then be started any number
of times.  When the VM shuts down, the VMM exits, so if the VM is
started again after that, run-vmm will be run again with the same VM ID.
If this were not the case, VM IDs could not be assigned at import time,
and stay the same between successive boots of the same VM.

If your cgroup is not being emptied as expected, that is a bug,
regardless of whether it ends up being used again, because there
shouldn't be lingering processes or cgroups hanging around forever.
It sounds like we should have a test that confirms that this doesn't
happen, if it's hard to get right.

The networking test does do something slightly unusual, which is that it
restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
Hypervisor API.  I would expect that this runs the finish script, but
might be worth confirming.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 1572 bytes --]

On 7/16/26 04:29, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> The networking test runs run-vmm twice with the same VM ID.
>> If it waits for the previous cgroup to be empty, the second
>> call waits forever.
>>
>> Is this expected?  I would expect that run-vmm is only called
>> once.
> 
> Assuming you mean run twice in sequence, rather than run twice at the
> same time, then yes, it is expected.
> 
> The way it works at the moment is that imported VMs have a VM ID
> assigned when they're imported, and they can then be started any number
> of times.  When the VM shuts down, the VMM exits, so if the VM is
> started again after that, run-vmm will be run again with the same VM ID.
> If this were not the case, VM IDs could not be assigned at import time,
> and stay the same between successive boots of the same VM.
> 
> If your cgroup is not being emptied as expected, that is a bug,
> regardless of whether it ends up being used again, because there
> shouldn't be lingering processes or cgroups hanging around forever.
> It sounds like we should have a test that confirms that this doesn't
> happen, if it's hard to get right.
> 
> The networking test does do something slightly unusual, which is that it
> restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
> Hypervisor API.  I would expect that this runs the finish script, but
> might be worth confirming.

What should happen if run-vmm is run more than once for the same VM?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 7/16/26 04:29, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> The networking test runs run-vmm twice with the same VM ID.
>>> If it waits for the previous cgroup to be empty, the second
>>> call waits forever.
>>>
>>> Is this expected?  I would expect that run-vmm is only called
>>> once.
>> 
>> Assuming you mean run twice in sequence, rather than run twice at the
>> same time, then yes, it is expected.
>> 
>> The way it works at the moment is that imported VMs have a VM ID
>> assigned when they're imported, and they can then be started any number
>> of times.  When the VM shuts down, the VMM exits, so if the VM is
>> started again after that, run-vmm will be run again with the same VM ID.
>> If this were not the case, VM IDs could not be assigned at import time,
>> and stay the same between successive boots of the same VM.
>> 
>> If your cgroup is not being emptied as expected, that is a bug,
>> regardless of whether it ends up being used again, because there
>> shouldn't be lingering processes or cgroups hanging around forever.
>> It sounds like we should have a test that confirms that this doesn't
>> happen, if it's hard to get right.
>> 
>> The networking test does do something slightly unusual, which is that it
>> restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
>> Hypervisor API.  I would expect that this runs the finish script, but
>> might be worth confirming.
>
> What should happen if run-vmm is run more than once for the same VM?

At the same time?  That shouldn't happen.  That can either be ensured
outside of run-vmm or inside it — I don't think that every time we
factor out a common script it needs to add a new check that it's being
used correctly, or the codebase will end up all usage checks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 3196 bytes --]

On 7/16/26 04:29, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> The networking test runs run-vmm twice with the same VM ID.
>> If it waits for the previous cgroup to be empty, the second
>> call waits forever.
>>
>> Is this expected?  I would expect that run-vmm is only called
>> once.
> 
> Assuming you mean run twice in sequence, rather than run twice at the
> same time, then yes, it is expected.
> 
> The way it works at the moment is that imported VMs have a VM ID
> assigned when they're imported, and they can then be started any number
> of times.  When the VM shuts down, the VMM exits, so if the VM is
> started again after that, run-vmm will be run again with the same VM ID.
> If this were not the case, VM IDs could not be assigned at import time,
> and stay the same between successive boots of the same VM.
> 
> If your cgroup is not being emptied as expected, that is a bug,
> regardless of whether it ends up being used again, because there
> shouldn't be lingering processes or cgroups hanging around forever.
> It sounds like we should have a test that confirms that this doesn't
> happen, if it's hard to get right.
> 
> The networking test does do something slightly unusual, which is that it
> restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
> Hypervisor API.  I would expect that this runs the finish script, but
> might be worth confirming.

It does indeed run the finish script!

It appears there are likely two or more bugs.  The first is that
sending a signal to the run-vmm process itself is insufficient:
it leaves child processes behind.  I don't know why that is.  Is it
something you could figure out?

The second bug was that the finish script purged the wrong cgroup.
It tried to purge /vmm, which doesn't exist.  Purging a nonexistent
cgroup is a (intentionally) a no-op, so the finish script failed to
do anything.

Because of these two problems, there were still processes left in the
cgroup when run-vmm ran again.  Therefore, cgroup-setup was waiting
for them to exit, which never happened.

I haven't fixed the first problem, but I did fix the second.  However,
this caused /run/vsock/${1}/vsock to be left behind.  I suspect this
is because cgroup-setup sends SIGKILL, so Cloud Hypervisor never gets
to unlink the socket.  This caused Cloud Hypervisor to fail to restart.

To solve *that* problem, I had to add `rm -f /run/vsock/${1}/vsock`
to run-vmm.  Fortunately, this is safe against symlink attacks: the
VMM has no write access to /run/vsock/${1}, so it can't turn that
into a symlink, and rm doesn't follow symlinks in the file it is told
to delete.

Once I added the rm call, the tests passed!

This means that the Rust code was actually working fine, and that the
way it waits for processes to exit works properly.  The problem was
in the surrounding execline scripting.  While the Rust code *was*
missing locking, that was not the cause of any of the problems in
the test suite.

Do you have suggestions for fixing run-vmm and/or preventing similar
problems in the future?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 3080 bytes --]

On 7/16/26 16:14, Demi Marie Obenour wrote:
> On 7/16/26 04:29, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>
>>> The networking test runs run-vmm twice with the same VM ID.
>>> If it waits for the previous cgroup to be empty, the second
>>> call waits forever.
>>>
>>> Is this expected?  I would expect that run-vmm is only called
>>> once.
>>
>> Assuming you mean run twice in sequence, rather than run twice at the
>> same time, then yes, it is expected.
>>
>> The way it works at the moment is that imported VMs have a VM ID
>> assigned when they're imported, and they can then be started any number
>> of times.  When the VM shuts down, the VMM exits, so if the VM is
>> started again after that, run-vmm will be run again with the same VM ID.
>> If this were not the case, VM IDs could not be assigned at import time,
>> and stay the same between successive boots of the same VM.
>>
>> If your cgroup is not being emptied as expected, that is a bug,
>> regardless of whether it ends up being used again, because there
>> shouldn't be lingering processes or cgroups hanging around forever.
>> It sounds like we should have a test that confirms that this doesn't
>> happen, if it's hard to get right.
>>
>> The networking test does do something slightly unusual, which is that it
>> restarts the VMM with s6-svc -Wr -r, rather than e.g. via the Cloud
>> Hypervisor API.  I would expect that this runs the finish script, but
>> might be worth confirming.
> 
> It does indeed run the finish script!
> 
> It appears there are likely two or more bugs.  The first is that
> sending a signal to the run-vmm process itself is insufficient:
> it leaves child processes behind.  I don't know why that is.  Is it
> something you could figure out?
> 
> The second bug was that the finish script purged the wrong cgroup.
> It tried to purge /vmm, which doesn't exist.  Purging a nonexistent
> cgroup is a (intentionally) a no-op, so the finish script failed to
> do anything.
> 
> Because of these two problems, there were still processes left in the
> cgroup when run-vmm ran again.  Therefore, cgroup-setup was waiting
> for them to exit, which never happened.
> 
> I haven't fixed the first problem, but I did fix the second.  However,
> this caused /run/vsock/${1}/vsock to be left behind.  I suspect this
> is because cgroup-setup sends SIGKILL, so Cloud Hypervisor never gets
> to unlink the socket.  This caused Cloud Hypervisor to fail to restart.
> 
> To solve *that* problem, I had to add `rm -f /run/vsock/${1}/vsock`
> to run-vmm.  Fortunately, this is safe against symlink attacks: the
> VMM has no write access to /run/vsock/${1}, so it can't turn that
> into a symlink, and rm doesn't follow symlinks in the file it is told
> to delete.
> 
> Once I added the rm call, the tests passed!
It kept passing even after I removed the second --no-wait from run-vmm,
which was the workaround for the bug.  I should have done that before.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Networking test runs run-vmm twice with the same VM ID

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> It appears there are likely two or more bugs.  The first is that
> sending a signal to the run-vmm process itself is insufficient:
> it leaves child processes behind.  I don't know why that is.  Is it
> something you could figure out?

I would guess that this is because bwrap forks.  I noticed this the
other day.  I think the right solution for this, since we're not using
bwrap for anything complicated, is to replace it with individual
unshares, mounts, etc. that don't fork.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]