Re: virtio-msg inter-VM transport vs virtio-vhost-user

Edgar E. Iglesias
2026-03-04T16:01:38Z

On Tue, Mar 03, 2026 at 12:56:46PM -0500, Demi Marie Obenour wrote:
> Spectrum (https://spectrum-os.org) is going to be implementing
> virtio devices outside of the host. One proposed method of doing
> this is virtio-vhost-user, which is a virtio device that allows a
> VM to expose a vhost-user device to another VM. For instance, one
> could assign a NIC to one VM and have it provide a vhost-user-net
> device for use by a different VM.
>
> I brought this up on the KVM/QEMU community call today. Alex Bennée
> recommended using virtio-msg instead. However, I have a few concerns
> with this:
>
> 1. Virtio-msg buses are specific to a given hypervisor or (in the
> case of FF-A) to a given CPU architecture. None of the current
> buses support KVM on platforms other than Arm64. Therefore,
> a brand-new bus would be needed.

Hi Demi,

The virtio-msg AMP PCI bus works on KVM both for ARM and x86 and
any other arch, it's generic. These are the patches we posted to
qemu-devel:

https://lore.kernel.org/qemu-devel/20260224155721.612314-1-edgar.iglesias@gmail.com/

>
> 2. Virtio-msg requires not-yet-upstream drivers in both the frontend
> (the VM using the device) and the backend (the VM providing the
> device). Vhost-user uses any of the existing transports, such as
> PCI or MMIO. This means that upstream drivers can be used in the
> frontend, and also enables supports for Windows and other guests
> that lack support for virtio-msg.

Fair point, a bit of chicken and egg...

>
> 3. Vhost-user is already widely deployed, so frontend implementations
> are quite well tested. A KVM-specific virtio-msg transport would
> serve only one purpose: driver VMs (with assigned devices) on
> non-Arm64 platforms. This is a quite niche use-case. Therefore,
> I'm concerned that the needed frontend code will be poorly tested
> and bitrot.
>
> Manos Pitsidianakis stated that vhost-user does not make sense in
> this case. Why is that? Would it make sense to use virtio-msg
> between VMM and its VM, and expose a vhost-user device to the
> outside world? What about having the virtio-vhost-user guest driver
> emulate a virtio-msg transport, so that it can be used with any device
> implementation supporting virtio-msg?
>
> I would greatly appreciate any and all suggestions here. This is a
> serious project that is going to be used in production, but I want
> to ensure that the design is the best possible.

I've not looked in detail at virtio-vhost-user but it seems to be
a bit similar to virtio-msg over PCI AMP with some differences.

My take is:
vhost-user is an interface designed for splitting virtio transport
from virtio device backend in user-space on the same OS instance.
The virtio-vhost-user device, enables the use of vhost-user across
VM's streching vhost-user's intended use. This is a great step but
Since vhost-user is not a virtio transport, It is not end-to-end in
the sense that you need QEMU to translate virtio-pci to vhost-user and
tunnel it over the virtio-vhost-user device to the backend.

Virtio-msg is from the start a transport meant to work between
heterogenous systems, different OS, OS instances even different
SoCs. It's end-to-end in the sense that if you have a front-end
driver in the front-end VM with an appropriate virtio-msg bus you can
talk directly with the backend without intermediate proxies,
potentially without VM exits.

Cheers,
Edgar