virtio-msg inter-VM transport vs virtio-vhost-user

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 2133 bytes --]

Spectrum (https://spectrum-os.org) is going to be implementing
virtio devices outside of the host.  One proposed method of doing
this is virtio-vhost-user, which is a virtio device that allows a
VM to expose a vhost-user device to another VM.  For instance, one
could assign a NIC to one VM and have it provide a vhost-user-net
device for use by a different VM.

I brought this up on the KVM/QEMU community call today.  Alex Bennée
recommended using virtio-msg instead.  However, I have a few concerns
with this:

1. Virtio-msg buses are specific to a given hypervisor or (in the
   case of FF-A) to a given CPU architecture.  None of the current
   buses support KVM on platforms other than Arm64.  Therefore,
   a brand-new bus would be needed.

2. Virtio-msg requires not-yet-upstream drivers in both the frontend
   (the VM using the device) and the backend (the VM providing the
   device).  Vhost-user uses any of the existing transports, such as
   PCI or MMIO.  This means that upstream drivers can be used in the
   frontend, and also enables supports for Windows and other guests
   that lack support for virtio-msg.

3. Vhost-user is already widely deployed, so frontend implementations
   are quite well tested.  A KVM-specific virtio-msg transport would
   serve only one purpose: driver VMs (with assigned devices) on
   non-Arm64 platforms.  This is a quite niche use-case.  Therefore,
   I'm concerned that the needed frontend code will be poorly tested
   and bitrot.

Manos Pitsidianakis stated that vhost-user does not make sense in
this case.  Why is that?  Would it make sense to use virtio-msg
between VMM and its VM, and expose a vhost-user device to the
outside world?  What about having the virtio-vhost-user guest driver
emulate a virtio-msg transport, so that it can be used with any device
implementation supporting virtio-msg?

I would greatly appreciate any and all suggestions here.  This is a
serious project that is going to be used in production, but I want
to ensure that the design is the best possible.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]