Re: virtio-msg inter-VM transport vs virtio-vhost-user

[Bertrand Marquis <Bertrand.Marquis@arm.com>]

Hi Demi,

> On 3 Mar 2026, at 18:56, Demi Marie Obenour <demiobenour@gmail.com> wrote:
>
> Spectrum (https://spectrum-os.org) is going to be implementing
> virtio devices outside of the host.  One proposed method of doing
> this is virtio-vhost-user, which is a virtio device that allows a
> VM to expose a vhost-user device to another VM.  For instance, one
> could assign a NIC to one VM and have it provide a vhost-user-net
> device for use by a different VM.
>
> I brought this up on the KVM/QEMU community call today.  Alex Bennée
> recommended using virtio-msg instead.  However, I have a few concerns
> with this:
>
> 1. Virtio-msg buses are specific to a given hypervisor or (in the
>   case of FF-A) to a given CPU architecture.  None of the current
>   buses support KVM on platforms other than Arm64.  Therefore,
>   a brand-new bus would be needed.

Even FF-A is not useable at the moment with KVM as there is no FF-A
support for VM to VM in KVM (only host can communicate with the secure
world).

MMIO or PCI based virtio in KVM case is working and was not really
the target of our work.

pKVM is a target and is being worked on using FF-A but pVM to Host
virtio is still using PCI at the moment

Now creating a KVM specific bus reusing the concept of a FIFO
to transfer the messages between a VM and Host is definitely possible
to do and should not be that complex.

Right now i am working on backend implementation where:
- the bus implementation would be in linux kernel allowing several
implementations like FF-A, xen or others to be done as linux drivers.
- have a bus interface provided to user land so that Qemu could contain
the transport implementation but would not need to be modified for new
bus implementations.

Reusing this, it should be fairly simple to define a KVM bus and reuse
the other parts of the implementations.

>
> 2. Virtio-msg requires not-yet-upstream drivers in both the frontend
>   (the VM using the device) and the backend (the VM providing the
>   device).  Vhost-user uses any of the existing transports, such as
>   PCI or MMIO.  This means that upstream drivers can be used in the
>   frontend, and also enables supports for Windows and other guests
>   that lack support for virtio-msg.

This is definitely true and will always stay true. To use virtio-msg you
will need a new transport implementation for it when you want to use
it and the bus implementation(s) you require. To be used in windows
those part will also be needed.

In your example here you rely on existing MMIO or PCI transport and
existing vhost implementations. This is definitely quicker to do and use.
The goal is not replace what works but to provide solutions for use cases
where MMIO or PCI currently do not work or need to be optimized.

>
> 3. Vhost-user is already widely deployed, so frontend implementations
>   are quite well tested.  A KVM-specific virtio-msg transport would
>   serve only one purpose: driver VMs (with assigned devices) on
>   non-Arm64 platforms.  This is a quite niche use-case.  Therefore,
>   I'm concerned that the needed frontend code will be poorly tested
>   and bitrot.

We are in the process of defining the specification for virtio-msg and we
are working on implementations in parallel so our implementations are
for now not widely tested that is clear.
Now a specific KVM virtio message bus implementation would reuse
the transport and driver implementations which would be used on any
platforms in the future. I am not following your niche use-case here
and the poorly tested argument. Maybe i am missing something.

>
> Manos Pitsidianakis stated that vhost-user does not make sense in
> this case.  Why is that?  Would it make sense to use virtio-msg
> between VMM and its VM, and expose a vhost-user device to the
> outside world?  What about having the virtio-vhost-user guest driver
> emulate a virtio-msg transport, so that it can be used with any device
> implementation supporting virtio-msg?

I am not following your point here. You want to do virtio on top of virtio ?

Regards
Bertrand

>
> I would greatly appreciate any and all suggestions here.  This is a
> serious project that is going to be used in production, but I want
> to ensure that the design is the best possible.
> --
> Sincerely,
> Demi Marie Obenour (she/her/hers)
> <OpenPGP_0xB288B55FFF9C22C1.asc>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.