From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id E3101AD9F; Thu, 28 May 2026 16:13:26 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 993) id 0A224AD84; Thu, 28 May 2026 16:13:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-0.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_PASS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=4.0.1 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by atuin.qyliss.net (Postfix) with ESMTPS id 7CC9BAD83 for ; Thu, 28 May 2026 16:13:22 +0000 (UTC) Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-45eee266c6cso277519f8f.1 for ; Thu, 28 May 2026 09:13:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1779984797; x=1780589597; darn=spectrum-os.org; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=JWT+VWYM1XqareEencxyOMF9mKtVF1ZIOZBhiJl2yJ8=; b=U99lvKsy4aGyk9IOkk2i2hIYWvhPR8vrB0GOhbRRgDfa0YH4ZW488QgZn+YSmgWCC6 junZbag8zjghe1Z7vm09zYFr74OVVa4QXI9DckdFo+j0MxF40g2AxhCJ7pxGbrgcvWF5 Q1ROB7WtEHmPfcgYv9ykyK8iO+xBe2gX3udDi3GfRGy+zaa4Vx3fEL6gptGsbpWDoA68 UACSO5PF/j1fIV9u9qYn2g1223OtV1Ad+Xs61RkDBXtNMVu4i2ntxcymEMcpJYCOf2/M mMra+xMw88ntYrddjuT/bhdGJ7eBaJPpAmsskqs9+yclOnQ848QQ6arBQWUFNe4Tggsa LFUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779984797; x=1780589597; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=JWT+VWYM1XqareEencxyOMF9mKtVF1ZIOZBhiJl2yJ8=; b=pTn4ySgsfz7+GWx5uXxHTz9eekIJcaU3JcZMFkxf6OnW2LVop9L9vbXDcoRRcEkLhU i7zLgZCam8bIS859cQl4PFqY1/1ibwBa86IGC1rkuKvjpRJ6CBFi5tRfBSis55tpCvj4 /qT5wOHB79tLfzK/dDcZkKUxZDcYcCOBZOMUD+dcbfIxSIveHyTwE0NcAsX3BeRQX3hW mC+qnDJFQEngUluP4DSc6tC+gfscyJUNc/VSqBkqEVlFSMRrfhFjv46Ii6zl3rj6MaGW QIg3riiy1JGJy+2QgDWlq0LIfJQBiayt9oert+Lw+JHyJTCGU58OQwOKZ6bCYHxC1zqO rLlg== X-Forwarded-Encrypted: i=1; AFNElJ9bUFQrm/Nl4eM11TOMziHxWhqHzRw3c83B0SVGqYIMJ5p9nLlxpEw3o6gyCtz3Y4+VsCPz8g==@spectrum-os.org X-Gm-Message-State: AOJu0YzgoWl+UmzxGtjn9Ipj49RYqzB5P2v5/zT/xR+pYLdIc3o7Fm8B BcOxOk4Ir9gOQi99CX+rc4Fp6qvNZ5wYGdKrSMOnDNMQyMghLeyzdNJiijO5tdbMV8U= X-Gm-Gg: Acq92OEkgaQnVHNGmdem0FCkO+EXu/KBoKQLPWm9Dd3hVj1LF+3WjHpq8rGRumaURaZ at+0mJtA2mZm5rEzOnnYi5IFHJckFdpjhiikuQ/KOW1iWD9fPdbytDhHsaLKyamkmbMLC+bXo9T oiigBmZMJTUIidevRQC2EGWQ7G5ksonwzWyldbZdIdG6RBpbD36xSiSJuoJrcAwaq8q23zz7KEP iHamXf2yNhxWXlzidgE6WOpSWYmtA6EcuXC3F7CPVUXiW2UjiSGr0RW6oa4r1EjO0DU5P0JxS1X AUYPL4kUuYZtSsQovZYtd5ZloA0RzpXVLabfrNsF0ArBPQE5wVrsuGyYxlM1K8trSHI61QhZ6ro cIi/HTxvkX07YiXutQjYuRcH0h3UkuPwgOaxNtiqR4CXrEXV0xjg7iRbErbezjtiva1aBCd/1sk 6Fvq64NaJbC8dYA2a4vCo9gCD6UCQVmesXxzgsAtZciA+q X-Received: by 2002:a05:6000:2883:b0:43c:f66e:f24 with SMTP id ffacd0b85a97d-45eb38c8b80mr48858338f8f.35.1779984796735; Thu, 28 May 2026 09:13:16 -0700 (PDT) Received: from draig.lan ([185.124.0.195]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-45edb5579b8sm14740521f8f.9.2026.05.28.09.13.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 May 2026 09:13:15 -0700 (PDT) Received: from draig (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id C9E555F92B; Thu, 28 May 2026 17:13:14 +0100 (BST) From: =?utf-8?Q?Alex_Benn=C3=A9e?= To: Demi Marie Obenour Subject: Re: Vhost-guest (was virtio vhost-user) vs virtio-msg In-Reply-To: <4ba67246-3a2b-43ee-b827-000c16ff07ed@gmail.com> (Demi Marie Obenour's message of "Thu, 28 May 2026 02:59:13 -0400") References: <11771164-7919-43e1-a980-03f036bdae2e@gmail.com> <4ba67246-3a2b-43ee-b827-000c16ff07ed@gmail.com> User-Agent: mu4e 1.14.1; emacs 30.1 Date: Thu, 28 May 2026 17:13:03 +0100 Message-ID: <877bonqzqo.fsf@draig.linaro.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Message-ID-Hash: RBVRV7P7X56SMWD6PJSRLMROZ2T73PV2 X-Message-ID-Hash: RBVRV7P7X56SMWD6PJSRLMROZ2T73PV2 X-MailFrom: alex.bennee@linaro.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; header-match-devel.spectrum-os.org-4; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Manos Pitsidianakis , Parav Pandit , Alyssa Ross , Rob Bradford , Bo Chen , "dev@lists.cloudhypervisor.org" , Spectrum OS Development , "virtio-comment@lists.linux.dev" , Viresh Kumar , Sumit Semwal , Bill Mills , "Edgar E. Iglesias" X-Mailman-Version: 3.3.9 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Demi Marie Obenour writes: (add virtio-msg authors to CC) > On 5/28/26 01:47, Manos Pitsidianakis wrote: >> On Thu, May 28, 2026 at 8:22=E2=80=AFAM Parav Pandit = wrote: >>> >>> >>>> From: Demi Marie Obenour >>>> Sent: 28 May 2026 05:23 AM >>>> To: virtio-comment@lists.linux.dev >>>> Subject: MSI-X vector limits and reserving a virtio device ID >>>> >>>> I'd like to reserve a virtio device ID for virtio vhost-guest, >>>> formally virtio vhost-user. Would this be possible? >>>> >>> Vhost user is an implementation of the device. >>> I believe it stays as implementation and not a new device type. >>=20 >> This exactly. >>=20 >> Furthermore, we already have a mechanism for "providing" an arbitrary >> virtio device; it's called a transport. >>=20 >> Demi, I suggest you look into virtio-msg transport, which would allow >> you to do what you want. > > I'm aware of virtio-msg, and in fact considered using it. However, I > found that virtio-msg doesn't meet my requirements. > > 1. Virtio-msg needs to run over a transport of its own. None of the > proposed transports support KVM guests on x86. FF-A is the only > one that would make sense with KVM, but FF-A is specific to Arm. Well yes. virtio-msg is a common transport that implements various buses on the backend. FF-A is one but we have working implementations that just use plain memory sharing for the message bus which work on anything including x86/KVM. Although it does beg the question of what an additional transport would being to KVM as it is already well served by PCI and MMIO transports. > 2. Virtio-msg isn't compatible with existing frontend VMMs. > Vhost-guest can be used with any frontend VMM that implements > vhost-user. What exactly do you mean by frontend VMMs? The VMM needs some mechanism to expose a VirtIO transport to the guest be it through probing (PCI) or some machine description like ACPI or DT. The guest is completely unaware if the backend implementation is using vhost-user. If you were going to expose that to the guest you will need some mechanism for that. For what its worth there are QEMU and rust-vmm implementations of various virtio-msg backends although we would expect the host UAPI to be stabilised after the VirtIO spec is ratified (because its not guest visible). > 3. Virtio-msg isn't compatible with existing frontend drivers. While I > expect that drivers for Linux will eventually be upstreamed, This is just plain wrong. No changes are needed to be made to the drivers as they are transport agnostic. > I doubt that drivers for Windows or *BSD will ever be written. > I don't even know if the Windows Driver Framework provides enough > to write one. I don't need this myself, but I suspect that this > is enough to make virtio-msg unsuitable in many environments. Why would Windows or *BSD want to implement virtio-msg when they can already use MMIO and PCI? But nothing stops them implementing it if they wish. >=20=20=20=20 > 4. Virtio-msg requires invasive changes to existing userspace device > implementations. No it doesn't. We test virtio-msg with existing unmodified rust-vmm vhost-device implementations because on the host we bridge between virtio-msg and vhost-user. In QEMU the transport is abstracted away from the details of the device implementation - you don't need MMIO and PCI specific device implementations either. > The project I work on doesn't use QEMU, and the > existing frontends are targeted at server use-cases. Using the > vhost-user protocol lets me reuse these with little effort. > > 5. To the best of my knowledge, virtio-msg doesn't support live > migration of frontend VMs. Vhost-guest uses the vhost-user > protocol, which has supported this for a very long time. I don't > need live migration myself, but for many server use-cases, not > having live migration is a dealbreaker. Live migration isn't in scope for a transport (aside from maybe support device reset/disable flows). The information needed to deal with migration is between the VMM and whatever implements the device backend. Indeed I find it a little confusing how live migration would work if the vhost-guest communication is directly between the backend and the guest. The VMM is the one that is responsible for serialistion and if it is cut out of the loop how will it know? >=20=20=20=20 > 6. I don't know if virtio-msg can achieve comparable performance. > It appears to be optimized for reliability and isolation, > not processing tens of gigabits of network traffic per second. > Vhost-guest is designed with performance in mind. This is pure supposition. The data plane in virtio-msg is the same as in PCI and MMIO, shared memory and virtqs. While virtio-msg does support notifications in the message queue it does not preclude direct IRQ signalling or indeed switch to pure polling which is what most of the high speed networking solutions end up doing to avoid the latency of IRQs. > > Vhost-guest is designed for storage and networking appliances on > servers, whereas virtio-msg is designed for safety-critical embedded > systems. These domains have very different requirements, and as a > result they arrived at very different solutions. > > I work on Spectrum (https://spectrum-os.org), which uses Cloud > Hypervisor. As the name implies, Cloud Hypervisor is primarily > intended for cloud workloads, though it can also be used on clients. > I don't think that an implementation of virtio-msg would be accepted, > as it benefits none of Cloud Hypervisor's other users. =2D-=20 Alex Benn=C3=A9e Virtualisation Tech Lead @ Linaro --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFLBAEBCgA1FiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmoYaZoXHGFsZXguYmVu bmVlQGxpbmFyby5vcmcACgkQ+9DbCVqeKkTk4wf6AokfXLzNP6PWxrGjTsU19W5x MEayiyyE+2dZOA/vHD6w2qPUFW/HpduURw9egu05NEK4rTSSTPvUMRRuXC6gvesW UvuuGFHqOeAPZJeG97VvH8cWZ+Tqei23x2onrV0URZZ5Tv6ZXFApp41BsAPtpNuE fay4P/nbYWBgYdbSjLvrcEw+yNCpY4h+do3dRYhHR1oYYoAD8v2QKuMVGMzm6HUS fLP/g7re1gWck0P4cXrx+4E2GPh6ofI7EEkiyZS7s55jBcdZRq+1O99Jmduj65G3 /SBiGf3bBOg02fX+qwhO45wyPVj8j2DTtHzMFcqYJ3OEqRdaTK4MJDNIfMVh7A== =mT+u -----END PGP SIGNATURE----- --=-=-=--