Re: Vhost-guest (was virtio vhost-user) vs virtio-msg

[Alyssa Ross <hi@alyssa.is>]

[-- Attachment #1: Type: text/plain, Size: 5878 bytes --]

On Thu, May 28, 2026 at 02:59:13AM -0400, Demi Marie Obenour wrote:
> On 5/28/26 01:47, Manos Pitsidianakis wrote:
> > On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
> >>
> >>
> >>> From: Demi Marie Obenour <demiobenour@gmail.com>
> >>> Sent: 28 May 2026 05:23 AM
> >>> To: virtio-comment@lists.linux.dev
> >>> Subject: MSI-X vector limits and reserving a virtio device ID
> >>>
> >>> I'd like to reserve a virtio device ID for virtio vhost-guest,
> >>> formally virtio vhost-user.  Would this be possible?
> >>>
> >> Vhost user is an implementation of the device.
> >> I believe it stays as implementation and not a new device type.
> >
> > This exactly.
> >
> > Furthermore, we already have a mechanism for "providing" an arbitrary
> > virtio device; it's called a transport.
> >
> > Demi, I suggest you look into virtio-msg transport, which would allow
> > you to do what you want.
>
> I'm aware of virtio-msg, and in fact considered using it.  However, I
> found that virtio-msg doesn't meet my requirements.
>
> 1. Virtio-msg needs to run over a transport of its own.  None of the
>    proposed transports support KVM guests on x86.  FF-A is the only
>    one that would make sense with KVM, but FF-A is specific to Arm.
>
> 2. Virtio-msg isn't compatible with existing frontend VMMs.
>    Vhost-guest can be used with any frontend VMM that implements
>    vhost-user.
>
> 3. Virtio-msg isn't compatible with existing frontend drivers.  While I
>    expect that drivers for Linux will eventually be upstreamed,
>    I doubt that drivers for Windows or *BSD will ever be written.
>    I don't even know if the Windows Driver Framework provides enough
>    to write one.  I don't need this myself, but I suspect that this
>    is enough to make virtio-msg unsuitable in many environments.
>
> 4. Virtio-msg requires invasive changes to existing userspace device
>    implementations.  The project I work on doesn't use QEMU, and the
>    existing frontends are targeted at server use-cases.  Using the
>    vhost-user protocol lets me reuse these with little effort.
>
> 5. To the best of my knowledge, virtio-msg doesn't support live
>    migration of frontend VMs.  Vhost-guest uses the vhost-user
>    protocol, which has supported this for a very long time.  I don't
>    need live migration myself, but for many server use-cases, not
>    having live migration is a dealbreaker.
>
> 6. I don't know if virtio-msg can achieve comparable performance.
>    It appears to be optimized for reliability and isolation,
>    not processing tens of gigabits of network traffic per second.
>    Vhost-guest is designed with performance in mind.
>
> Vhost-guest is designed for storage and networking appliances on
> servers, whereas virtio-msg is designed for safety-critical embedded
> systems.  These domains have very different requirements, and as a
> result they arrived at very different solutions.
>
> I work on Spectrum (https://spectrum-os.org), which uses Cloud
> Hypervisor.  As the name implies, Cloud Hypervisor is primarily
> intended for cloud workloads, though it can also be used on clients.
> I don't think that an implementation of virtio-msg would be accepted,
> as it benefits none of Cloud Hypervisor's other users.

It would benefit them just as much as a virtio-vhost-user device would,
surely?  (And that's not a great justification from a spec point of view
regardless…)

But I think it's worth stepping back here and providing some context.
virtio-vhost-user is an idea that's been kicking around in various forms
since 2015, with incomplete implementations for QEMU, crosvm, and DPDK.
It's been picked up by a number of different people during that time,
and to my knowledge has only failed to make it across the line due to
lack of investment.  There's clear demand for a way to run a vhost-user
device inside a VM.

Links to previous iterations:

• Discussions about VVU vs. virtio-msg:
  https://lore.kernel.org/virtio-comment/b291c8af-58ee-46e4-9033-e681ff9677db@gmail.com/
  https://lore.kernel.org/virtio-comment/89d13644-04ac-4057-b197-a6efe5e82ef6@gmail.com/

• Inquiry into the possibility of this functionality:
  https://lore.kernel.org/virtualization/ff2b81b0-04ba-49b7-8a4e-089f1c875744@suse.com/

• Most recent previous draft VIRTIO spec submission, with links to
  implementations:
  https://lore.kernel.org/virtio-dev/20221007165643.3920613-1-usama.arif@bytedance.com/raw

• Another inquiry about how to do it:
  https://lore.kernel.org/qemu-devel/CADx_CBPzAstC0o9X6CrnyFqYYAtPbw5-XHWxmXTt6+LyYb-U3g@mail.gmail.com/

• Implementation in crosvm:
  https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3146213

• My own initial inquiry about it:
  https://lore.kernel.org/qemu-devel/87h7u1s5k1.fsf@alyssa.is/

• The last of another round of spec submissions:
  https://lore.kernel.org/virito-dev/1561418997-24608-1-git-send-email-ndragazis@arrikto.com/

(and more going back further)

My point here is that VVU has been the go-to answer for how this problem
could be solved for a very long time, during which an objection to a
VIRTIO device used to implement other VIRTIO devices hasn't been
widespread.  I'd be interested to understand what's changed there, if
indeed anything has.  Is it just that virtio-msg now exists?
Transporting VIRTIO over a stream isn't a huge leap, but the
participants in prior discussions didn't suggest it as an alternative.
From where I stand, virtio-msg seems at best an unproven and far-off way
to accomplish this, whereas VVU has proven to work, at least well
enough to have been considered worth putting in work to try to move
the spec forward for multiple different (as far as I know) independent
efforts over the years.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]