Vhost-guest (was virtio vhost-user) vs virtio-msg

Vhost-guest (was virtio vhost-user) vs virtio-msg

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 3195 bytes --]

On 5/28/26 01:47, Manos Pitsidianakis wrote:
> On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
>>
>>
>>> From: Demi Marie Obenour <demiobenour@gmail.com>
>>> Sent: 28 May 2026 05:23 AM
>>> To: virtio-comment@lists.linux.dev
>>> Subject: MSI-X vector limits and reserving a virtio device ID
>>>
>>> I'd like to reserve a virtio device ID for virtio vhost-guest,
>>> formally virtio vhost-user.  Would this be possible?
>>>
>> Vhost user is an implementation of the device.
>> I believe it stays as implementation and not a new device type.
> 
> This exactly.
> 
> Furthermore, we already have a mechanism for "providing" an arbitrary
> virtio device; it's called a transport.
> 
> Demi, I suggest you look into virtio-msg transport, which would allow
> you to do what you want.

I'm aware of virtio-msg, and in fact considered using it.  However, I
found that virtio-msg doesn't meet my requirements.

1. Virtio-msg needs to run over a transport of its own.  None of the
   proposed transports support KVM guests on x86.  FF-A is the only
   one that would make sense with KVM, but FF-A is specific to Arm.

2. Virtio-msg isn't compatible with existing frontend VMMs.
   Vhost-guest can be used with any frontend VMM that implements
   vhost-user.

3. Virtio-msg isn't compatible with existing frontend drivers.  While I
   expect that drivers for Linux will eventually be upstreamed,
   I doubt that drivers for Windows or *BSD will ever be written.
   I don't even know if the Windows Driver Framework provides enough
   to write one.  I don't need this myself, but I suspect that this
   is enough to make virtio-msg unsuitable in many environments.
   
4. Virtio-msg requires invasive changes to existing userspace device
   implementations.  The project I work on doesn't use QEMU, and the
   existing frontends are targeted at server use-cases.  Using the
   vhost-user protocol lets me reuse these with little effort.

5. To the best of my knowledge, virtio-msg doesn't support live
   migration of frontend VMs.  Vhost-guest uses the vhost-user
   protocol, which has supported this for a very long time.  I don't
   need live migration myself, but for many server use-cases, not
   having live migration is a dealbreaker.
   
6. I don't know if virtio-msg can achieve comparable performance.
   It appears to be optimized for reliability and isolation,
   not processing tens of gigabits of network traffic per second.
   Vhost-guest is designed with performance in mind.

Vhost-guest is designed for storage and networking appliances on
servers, whereas virtio-msg is designed for safety-critical embedded
systems.  These domains have very different requirements, and as a
result they arrived at very different solutions.

I work on Spectrum (https://spectrum-os.org), which uses Cloud
Hypervisor.  As the name implies, Cloud Hypervisor is primarily
intended for cloud workloads, though it can also be used on clients.
I don't think that an implementation of virtio-msg would be accepted,
as it benefits none of Cloud Hypervisor's other users.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Vhost-guest (was virtio vhost-user) vs virtio-msg

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 5878 bytes --]

On Thu, May 28, 2026 at 02:59:13AM -0400, Demi Marie Obenour wrote:
> On 5/28/26 01:47, Manos Pitsidianakis wrote:
> > On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
> >>
> >>
> >>> From: Demi Marie Obenour <demiobenour@gmail.com>
> >>> Sent: 28 May 2026 05:23 AM
> >>> To: virtio-comment@lists.linux.dev
> >>> Subject: MSI-X vector limits and reserving a virtio device ID
> >>>
> >>> I'd like to reserve a virtio device ID for virtio vhost-guest,
> >>> formally virtio vhost-user.  Would this be possible?
> >>>
> >> Vhost user is an implementation of the device.
> >> I believe it stays as implementation and not a new device type.
> >
> > This exactly.
> >
> > Furthermore, we already have a mechanism for "providing" an arbitrary
> > virtio device; it's called a transport.
> >
> > Demi, I suggest you look into virtio-msg transport, which would allow
> > you to do what you want.
>
> I'm aware of virtio-msg, and in fact considered using it.  However, I
> found that virtio-msg doesn't meet my requirements.
>
> 1. Virtio-msg needs to run over a transport of its own.  None of the
>    proposed transports support KVM guests on x86.  FF-A is the only
>    one that would make sense with KVM, but FF-A is specific to Arm.
>
> 2. Virtio-msg isn't compatible with existing frontend VMMs.
>    Vhost-guest can be used with any frontend VMM that implements
>    vhost-user.
>
> 3. Virtio-msg isn't compatible with existing frontend drivers.  While I
>    expect that drivers for Linux will eventually be upstreamed,
>    I doubt that drivers for Windows or *BSD will ever be written.
>    I don't even know if the Windows Driver Framework provides enough
>    to write one.  I don't need this myself, but I suspect that this
>    is enough to make virtio-msg unsuitable in many environments.
>
> 4. Virtio-msg requires invasive changes to existing userspace device
>    implementations.  The project I work on doesn't use QEMU, and the
>    existing frontends are targeted at server use-cases.  Using the
>    vhost-user protocol lets me reuse these with little effort.
>
> 5. To the best of my knowledge, virtio-msg doesn't support live
>    migration of frontend VMs.  Vhost-guest uses the vhost-user
>    protocol, which has supported this for a very long time.  I don't
>    need live migration myself, but for many server use-cases, not
>    having live migration is a dealbreaker.
>
> 6. I don't know if virtio-msg can achieve comparable performance.
>    It appears to be optimized for reliability and isolation,
>    not processing tens of gigabits of network traffic per second.
>    Vhost-guest is designed with performance in mind.
>
> Vhost-guest is designed for storage and networking appliances on
> servers, whereas virtio-msg is designed for safety-critical embedded
> systems.  These domains have very different requirements, and as a
> result they arrived at very different solutions.
>
> I work on Spectrum (https://spectrum-os.org), which uses Cloud
> Hypervisor.  As the name implies, Cloud Hypervisor is primarily
> intended for cloud workloads, though it can also be used on clients.
> I don't think that an implementation of virtio-msg would be accepted,
> as it benefits none of Cloud Hypervisor's other users.

It would benefit them just as much as a virtio-vhost-user device would,
surely?  (And that's not a great justification from a spec point of view
regardless…)

But I think it's worth stepping back here and providing some context.
virtio-vhost-user is an idea that's been kicking around in various forms
since 2015, with incomplete implementations for QEMU, crosvm, and DPDK.
It's been picked up by a number of different people during that time,
and to my knowledge has only failed to make it across the line due to
lack of investment.  There's clear demand for a way to run a vhost-user
device inside a VM.

Links to previous iterations:

• Discussions about VVU vs. virtio-msg:
  https://lore.kernel.org/virtio-comment/b291c8af-58ee-46e4-9033-e681ff9677db@gmail.com/
  https://lore.kernel.org/virtio-comment/89d13644-04ac-4057-b197-a6efe5e82ef6@gmail.com/

• Inquiry into the possibility of this functionality:
  https://lore.kernel.org/virtualization/ff2b81b0-04ba-49b7-8a4e-089f1c875744@suse.com/

• Most recent previous draft VIRTIO spec submission, with links to
  implementations:
  https://lore.kernel.org/virtio-dev/20221007165643.3920613-1-usama.arif@bytedance.com/raw

• Another inquiry about how to do it:
  https://lore.kernel.org/qemu-devel/CADx_CBPzAstC0o9X6CrnyFqYYAtPbw5-XHWxmXTt6+LyYb-U3g@mail.gmail.com/

• Implementation in crosvm:
  https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3146213

• My own initial inquiry about it:
  https://lore.kernel.org/qemu-devel/87h7u1s5k1.fsf@alyssa.is/

• The last of another round of spec submissions:
  https://lore.kernel.org/virito-dev/1561418997-24608-1-git-send-email-ndragazis@arrikto.com/

(and more going back further)

My point here is that VVU has been the go-to answer for how this problem
could be solved for a very long time, during which an objection to a
VIRTIO device used to implement other VIRTIO devices hasn't been
widespread.  I'd be interested to understand what's changed there, if
indeed anything has.  Is it just that virtio-msg now exists?
Transporting VIRTIO over a stream isn't a huge leap, but the
participants in prior discussions didn't suggest it as an alternative.
From where I stand, virtio-msg seems at best an unproven and far-off way
to accomplish this, whereas VVU has proven to work, at least well
enough to have been considered worth putting in work to try to move
the spec forward for multiple different (as far as I know) independent
efforts over the years.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Vhost-guest (was virtio vhost-user) vs virtio-msg

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6623 bytes --]

On Thu, May 28, 2026 at 10:56:35AM +0200, Alyssa Ross wrote:
> On Thu, May 28, 2026 at 02:59:13AM -0400, Demi Marie Obenour wrote:
> > On 5/28/26 01:47, Manos Pitsidianakis wrote:
> > > On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
> > >>
> > >>
> > >>> From: Demi Marie Obenour <demiobenour@gmail.com>
> > >>> Sent: 28 May 2026 05:23 AM
> > >>> To: virtio-comment@lists.linux.dev
> > >>> Subject: MSI-X vector limits and reserving a virtio device ID
> > >>>
> > >>> I'd like to reserve a virtio device ID for virtio vhost-guest,
> > >>> formally virtio vhost-user.  Would this be possible?
> > >>>
> > >> Vhost user is an implementation of the device.
> > >> I believe it stays as implementation and not a new device type.
> > >
> > > This exactly.
> > >
> > > Furthermore, we already have a mechanism for "providing" an arbitrary
> > > virtio device; it's called a transport.
> > >
> > > Demi, I suggest you look into virtio-msg transport, which would allow
> > > you to do what you want.
> >
> > I'm aware of virtio-msg, and in fact considered using it.  However, I
> > found that virtio-msg doesn't meet my requirements.
> >
> > 1. Virtio-msg needs to run over a transport of its own.  None of the
> >    proposed transports support KVM guests on x86.  FF-A is the only
> >    one that would make sense with KVM, but FF-A is specific to Arm.
> >
> > 2. Virtio-msg isn't compatible with existing frontend VMMs.
> >    Vhost-guest can be used with any frontend VMM that implements
> >    vhost-user.
> >
> > 3. Virtio-msg isn't compatible with existing frontend drivers.  While I
> >    expect that drivers for Linux will eventually be upstreamed,
> >    I doubt that drivers for Windows or *BSD will ever be written.
> >    I don't even know if the Windows Driver Framework provides enough
> >    to write one.  I don't need this myself, but I suspect that this
> >    is enough to make virtio-msg unsuitable in many environments.
> >
> > 4. Virtio-msg requires invasive changes to existing userspace device
> >    implementations.  The project I work on doesn't use QEMU, and the
> >    existing frontends are targeted at server use-cases.  Using the
> >    vhost-user protocol lets me reuse these with little effort.
> >
> > 5. To the best of my knowledge, virtio-msg doesn't support live
> >    migration of frontend VMs.  Vhost-guest uses the vhost-user
> >    protocol, which has supported this for a very long time.  I don't
> >    need live migration myself, but for many server use-cases, not
> >    having live migration is a dealbreaker.
> >
> > 6. I don't know if virtio-msg can achieve comparable performance.
> >    It appears to be optimized for reliability and isolation,
> >    not processing tens of gigabits of network traffic per second.
> >    Vhost-guest is designed with performance in mind.
> >
> > Vhost-guest is designed for storage and networking appliances on
> > servers, whereas virtio-msg is designed for safety-critical embedded
> > systems.  These domains have very different requirements, and as a
> > result they arrived at very different solutions.
> >
> > I work on Spectrum (https://spectrum-os.org), which uses Cloud
> > Hypervisor.  As the name implies, Cloud Hypervisor is primarily
> > intended for cloud workloads, though it can also be used on clients.
> > I don't think that an implementation of virtio-msg would be accepted,
> > as it benefits none of Cloud Hypervisor's other users.
> 
> It would benefit them just as much as a virtio-vhost-user device would,
> surely?  (And that's not a great justification from a spec point of view
> regardless…)
> 
> But I think it's worth stepping back here and providing some context.
> virtio-vhost-user is an idea that's been kicking around in various forms
> since 2015, with incomplete implementations for QEMU, crosvm, and DPDK.
> It's been picked up by a number of different people during that time,
> and to my knowledge has only failed to make it across the line due to
> lack of investment.  There's clear demand for a way to run a vhost-user
> device inside a VM.
> 
> Links to previous iterations:
> 
> • Discussions about VVU vs. virtio-msg:
>   https://lore.kernel.org/virtio-comment/b291c8af-58ee-46e4-9033-e681ff9677db@gmail.com/
>   https://lore.kernel.org/virtio-comment/89d13644-04ac-4057-b197-a6efe5e82ef6@gmail.com/
> 
> • Inquiry into the possibility of this functionality:
>   https://lore.kernel.org/virtualization/ff2b81b0-04ba-49b7-8a4e-089f1c875744@suse.com/
> 
> • Most recent previous draft VIRTIO spec submission, with links to
>   implementations:
>   https://lore.kernel.org/virtio-dev/20221007165643.3920613-1-usama.arif@bytedance.com/raw
> 
> • Another inquiry about how to do it:
>   https://lore.kernel.org/qemu-devel/CADx_CBPzAstC0o9X6CrnyFqYYAtPbw5-XHWxmXTt6+LyYb-U3g@mail.gmail.com/
> 
> • Implementation in crosvm:
>   https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3146213
> 
> • My own initial inquiry about it:
>   https://lore.kernel.org/qemu-devel/87h7u1s5k1.fsf@alyssa.is/
> 
> • The last of another round of spec submissions:
>   https://lore.kernel.org/virito-dev/1561418997-24608-1-git-send-email-ndragazis@arrikto.com/
> 
> (and more going back further)
> 
> My point here is that VVU has been the go-to answer for how this problem
> could be solved for a very long time, during which an objection to a
> VIRTIO device used to implement other VIRTIO devices hasn't been
> widespread.  I'd be interested to understand what's changed there, if
> indeed anything has.  Is it just that virtio-msg now exists?
> Transporting VIRTIO over a stream isn't a huge leap, but the
> participants in prior discussions didn't suggest it as an alternative.
> From where I stand, virtio-msg seems at best an unproven and far-off way
> to accomplish this, whereas VVU has proven to work, at least well
> enough to have been considered worth putting in work to try to move
> the spec forward for multiple different (as far as I know) independent
> efforts over the years.

Both virtio-vhost-user and virtio-msg their own pros/cons and use cases.
Neither one comes out ahead in all cases, so I think it makes sense to
have both.

Demi, if you want to get the spec merged and are willing to participate
in maintenance (answering questions, reviewing spec extensions, etc)
then that would be nice. I will be around to transfer any knowledge I
still have of virtio-vhost-user and review the patches.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Vhost-guest (was virtio vhost-user) vs virtio-msg

[Alex Bennée]

[-- Attachment #1: Type: text/plain, Size: 5806 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

(add virtio-msg authors to CC)

> On 5/28/26 01:47, Manos Pitsidianakis wrote:
>> On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
>>>
>>>
>>>> From: Demi Marie Obenour <demiobenour@gmail.com>
>>>> Sent: 28 May 2026 05:23 AM
>>>> To: virtio-comment@lists.linux.dev
>>>> Subject: MSI-X vector limits and reserving a virtio device ID
>>>>
>>>> I'd like to reserve a virtio device ID for virtio vhost-guest,
>>>> formally virtio vhost-user.  Would this be possible?
>>>>
>>> Vhost user is an implementation of the device.
>>> I believe it stays as implementation and not a new device type.
>> 
>> This exactly.
>> 
>> Furthermore, we already have a mechanism for "providing" an arbitrary
>> virtio device; it's called a transport.
>> 
>> Demi, I suggest you look into virtio-msg transport, which would allow
>> you to do what you want.
>
> I'm aware of virtio-msg, and in fact considered using it.  However, I
> found that virtio-msg doesn't meet my requirements.
>
> 1. Virtio-msg needs to run over a transport of its own.  None of the
>    proposed transports support KVM guests on x86.  FF-A is the only
>    one that would make sense with KVM, but FF-A is specific to Arm.

Well yes. virtio-msg is a common transport that implements various buses
on the backend. FF-A is one but we have working implementations that
just use plain memory sharing for the message bus which work on anything
including x86/KVM. Although it does beg the question of what an
additional transport would being to KVM as it is already well served by
PCI and MMIO transports.

> 2. Virtio-msg isn't compatible with existing frontend VMMs.
>    Vhost-guest can be used with any frontend VMM that implements
>    vhost-user.

What exactly do you mean by frontend VMMs?

The VMM needs some mechanism to expose a VirtIO transport to the guest
be it through probing (PCI) or some machine description like ACPI or DT.
The guest is completely unaware if the backend implementation is using
vhost-user. If you were going to expose that to the guest you will need
some mechanism for that.

For what its worth there are QEMU and rust-vmm implementations of
various virtio-msg backends although we would expect the host UAPI to be
stabilised after the VirtIO spec is ratified (because its not guest
visible).

> 3. Virtio-msg isn't compatible with existing frontend drivers.  While I
>    expect that drivers for Linux will eventually be upstreamed,

This is just plain wrong. No changes are needed to be made to the
drivers as they are transport agnostic.

>    I doubt that drivers for Windows or *BSD will ever be written.
>    I don't even know if the Windows Driver Framework provides enough
>    to write one.  I don't need this myself, but I suspect that this
>    is enough to make virtio-msg unsuitable in many environments.

Why would Windows or *BSD want to implement virtio-msg when they can
already use MMIO and PCI? But nothing stops them implementing it if they
wish.

>    
> 4. Virtio-msg requires invasive changes to existing userspace device
>    implementations.

No it doesn't. We test virtio-msg with existing unmodified rust-vmm
vhost-device implementations because on the host we bridge between
virtio-msg and vhost-user. In QEMU the transport is abstracted away from
the details of the device implementation - you don't need MMIO and PCI
specific device implementations either.

>    The project I work on doesn't use QEMU, and the
>    existing frontends are targeted at server use-cases.  Using the
>    vhost-user protocol lets me reuse these with little effort.
>
> 5. To the best of my knowledge, virtio-msg doesn't support live
>    migration of frontend VMs.  Vhost-guest uses the vhost-user
>    protocol, which has supported this for a very long time.  I don't
>    need live migration myself, but for many server use-cases, not
>    having live migration is a dealbreaker.

Live migration isn't in scope for a transport (aside from maybe support
device reset/disable flows). The information needed to deal with
migration is between the VMM and whatever implements the device backend.

Indeed I find it a little confusing how live migration would work if the
vhost-guest communication is directly between the backend and the guest.
The VMM is the one that is responsible for serialistion and if it is cut
out of the loop how will it know?

>    
> 6. I don't know if virtio-msg can achieve comparable performance.
>    It appears to be optimized for reliability and isolation,
>    not processing tens of gigabits of network traffic per second.
>    Vhost-guest is designed with performance in mind.

This is pure supposition. The data plane in virtio-msg is the same as in
PCI and MMIO, shared memory and virtqs. While virtio-msg does support
notifications in the message queue it does not preclude direct IRQ
signalling or indeed switch to pure polling which is what most of the
high speed networking solutions end up doing to avoid the latency of
IRQs.

>
> Vhost-guest is designed for storage and networking appliances on
> servers, whereas virtio-msg is designed for safety-critical embedded
> systems.  These domains have very different requirements, and as a
> result they arrived at very different solutions.
>
> I work on Spectrum (https://spectrum-os.org), which uses Cloud
> Hypervisor.  As the name implies, Cloud Hypervisor is primarily
> intended for cloud workloads, though it can also be used on clients.
> I don't think that an implementation of virtio-msg would be accepted,
> as it benefits none of Cloud Hypervisor's other users.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: Vhost-guest (was virtio vhost-user) vs virtio-msg

[demiobenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 8990 bytes --]

On 5/28/26 12:13, Alex Bennée wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
> (add virtio-msg authors to CC)
> 
>> On 5/28/26 01:47, Manos Pitsidianakis wrote:
>>> On Thu, May 28, 2026 at 8:22 AM Parav Pandit <parav@nvidia.com> wrote:
>>>>
>>>>
>>>>> From: Demi Marie Obenour <demiobenour@gmail.com>
>>>>> Sent: 28 May 2026 05:23 AM
>>>>> To: virtio-comment@lists.linux.dev
>>>>> Subject: MSI-X vector limits and reserving a virtio device ID
>>>>>
>>>>> I'd like to reserve a virtio device ID for virtio vhost-guest,
>>>>> formally virtio vhost-user.  Would this be possible?
>>>>>
>>>> Vhost user is an implementation of the device.
>>>> I believe it stays as implementation and not a new device type.
>>>
>>> This exactly.
>>>
>>> Furthermore, we already have a mechanism for "providing" an arbitrary
>>> virtio device; it's called a transport.
>>>
>>> Demi, I suggest you look into virtio-msg transport, which would allow
>>> you to do what you want.
>>
>> I'm aware of virtio-msg, and in fact considered using it.  However, I
>> found that virtio-msg doesn't meet my requirements.
>>
>> 1. Virtio-msg needs to run over a transport of its own.  None of the
>>    proposed transports support KVM guests on x86.  FF-A is the only
>>    one that would make sense with KVM, but FF-A is specific to Arm.
> 
> Well yes. virtio-msg is a common transport that implements various buses
> on the backend. FF-A is one but we have working implementations that
> just use plain memory sharing for the message bus which work on anything
> including x86/KVM. Although it does beg the question of what an
> additional transport would being to KVM as it is already well served by
> PCI and MMIO transports.

My goal is to enable device implementations that run inside a VM,
rather than running on the host, for sandboxing reasons.  The PCI
and MMIO transports don't natively support this.  The purpose of
virtio-vhost-user is to add such support to the PCI transport.

Virtio-msg in KVM is definitely an alternative, but none of the
currently-proposed transports can be used as-is:

1. virtio-msg-ffa is specific to Arm.  Furthermore, the Xen
   implementation is not security supported, and I don't know if the
   pKVM implementation has been hardened against denial of service
   attacks.  The ones in Hafnium and TF-A were not last I checked.

2. virtio-msg-amp requires a (real or emulated) PCI connection.

3. virtio-msg-loopback is only within a single VM.

4. virtio-msg-xen is specific to Xen.

5. virtio-msg over admin virtqueues has not been implemented
   yet, whereas virtio-vhost-user has a nearly-complete device
   implementation.

Additionally, I would much prefer to reuse the existing vhost-user
protocol between frontend VMM and backend VMM.  This avoids having
to implement a third protocol on top of virtio-user and vfio-user.
Virtio-vhost-user maps to that protocol trivially (by design).

It should be possible to define and implement a virtio-msg binding
for virtio-vhost-user, though I have no particular plan to do so at
this time.

Additionally, virtio-vhost-user has more features than virtio-msg.
Some of these features are very useful, such as including the backend's
migration data in the frontend's migration stream.  Implementing them
in virtio-msg would be a nasty layering violation, whereas implementing
them in virtio-vhost-user is natural.

>> 2. Virtio-msg isn't compatible with existing frontend VMMs.
>>    Vhost-guest can be used with any frontend VMM that implements
>>    vhost-user.
> 
> What exactly do you mean by frontend VMMs?

The frontend VM is the VM that uses the virtio device.  The backend
VM is the VM that implements the device.  The frontend VMM is the
user-space code that runs the frontend VM, and the backend VMM is
the user-space code that runs the backend VM.

> The VMM needs some mechanism to expose a VirtIO transport to the guest
> be it through probing (PCI) or some machine description like ACPI or DT.
> The guest is completely unaware if the backend implementation is using
> vhost-user. If you were going to expose that to the guest you will need
> some mechanism for that.

With virtio-vhost-user, the frontend VMM runs a vhost-user client.
It exposes an MMIO or PCI device to its guest as it normally would.
The frontend VMM and its guest aren't aware of virtio-vhost-user.

> For what its worth there are QEMU and rust-vmm implementations of
> various virtio-msg backends although we would expect the host UAPI to be
> stabilised after the VirtIO spec is ratified (because its not guest
> visible).

Would you mind elaborating on "host UAPI"?  Do you mean "UAPI for
implementing a device"?

>> 3. Virtio-msg isn't compatible with existing frontend drivers.  While I
>>    expect that drivers for Linux will eventually be upstreamed,
> 
> This is just plain wrong. No changes are needed to be made to the
> drivers as they are transport agnostic.

I meant the exiting *transport* drivers.  

>>    I doubt that drivers for Windows or *BSD will ever be written.
>>    I don't even know if the Windows Driver Framework provides enough
>>    to write one.  I don't need this myself, but I suspect that this
>>    is enough to make virtio-msg unsuitable in many environments.
> 
> Why would Windows or *BSD want to implement virtio-msg when they can
> already use MMIO and PCI? But nothing stops them implementing it if they
> wish.

If VM A is going to expose a device to VM B using virtio-msg, VM B
needs virtio-msg support.  With virtio-vhost-user, VM B gets a MMIO
or PCI device.

>> 4. Virtio-msg requires invasive changes to existing userspace device
>>    implementations.
> 
> No it doesn't. We test virtio-msg with existing unmodified rust-vmm
> vhost-device implementations because on the host we bridge between
> virtio-msg and vhost-user. In QEMU the transport is abstracted away from
> the details of the device implementation - you don't need MMIO and PCI
> specific device implementations either.

Makes sense!  I was not aware of QEMU having that abstraction, but
it makes sense that it does.  rust-vmm really ought to also have
an abstraction, so that the same device implementation can support
virtio-vhost-user, vhost-msg, vhost-user, and in-process virtio.

>>    The project I work on doesn't use QEMU, and the
>>    existing frontends are targeted at server use-cases.  Using the
>>    vhost-user protocol lets me reuse these with little effort.
>>
>> 5. To the best of my knowledge, virtio-msg doesn't support live
>>    migration of frontend VMs.  Vhost-guest uses the vhost-user
>>    protocol, which has supported this for a very long time.  I don't
>>    need live migration myself, but for many server use-cases, not
>>    having live migration is a dealbreaker.
> 
> Live migration isn't in scope for a transport (aside from maybe support
> device reset/disable flows). The information needed to deal with
> migration is between the VMM and whatever implements the device backend.

Virtio-vhost-user has a different scope than virtio-msg.
Virtio-vhost-user lets you run a vhost-user server in a guest instead
of the host, so its scope includes everything that is in scope for
the vhost-user protocol.  That includes live migration.

> Indeed I find it a little confusing how live migration would work if the
> vhost-guest communication is directly between the backend and the guest.
> The VMM is the one that is responsible for serialistion and if it is cut
> out of the loop how will it know?

Live migration is a control plane operation.  On the sending side, the
backend VM sends migration data to its VMM over a dedicated virtqueue.
The backend VMM then forwards the migration data to the frontend VMM
over a pipe.  On the receiving side, the frontend VMM sends the data
to the backend VMM over a pipe, and the backend VMM sends it to the
backend VMM over a virtqueue.

This feature isn't in any of the currently published virtio-vhost-user
specifications, but the version I plan to submit to this list will
include it.

>> 6. I don't know if virtio-msg can achieve comparable performance.
>>    It appears to be optimized for reliability and isolation,
>>    not processing tens of gigabits of network traffic per second.
>>    Vhost-guest is designed with performance in mind.
> 
> This is pure supposition. The data plane in virtio-msg is the same as in
> PCI and MMIO, shared memory and virtqs. While virtio-msg does support
> notifications in the message queue it does not preclude direct IRQ
> signalling or indeed switch to pure polling which is what most of the
> high speed networking solutions end up doing to avoid the latency of
> IRQs.

Are shared memory and direct IRQ signalling an option for virtio over
admin virtqueues?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]