From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-4.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 26E202EB50; Wed, 17 Aug 2022 10:42:19 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id C55A52EBA9; Wed, 17 Aug 2022 10:42:16 +0000 (UTC) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by atuin.qyliss.net (Postfix) with ESMTPS id 4B5542EBA3 for ; Wed, 17 Aug 2022 10:42:13 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id A4E0D5C0091 for ; Wed, 17 Aug 2022 06:42:09 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 17 Aug 2022 06:42:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm3; t= 1660732929; x=1660819329; bh=/e+pY4F2y5e07H0P8Rm+Cgz01iTckno/oZx X5csVqDc=; b=bFe3N27d9RLNgGAVUnWdowB32bIszq3tL38to0nu4XrLccQTlia 66BmZ+9CRN/Zdkzl4WNqzoGOZwCVrmhw4o4ZuZ9kmmRbUBfAhs1G2dfPbNrSL5Mc rUD/B2N+pCeKDCZGoK1BO6sOs3ulc9StbJZT0jEABIWcoTXv8h/nBjMz8Cxnf6Uq PSvxU/69Cw7RXhihO+UWYBE5jOhaiqFq9tSW4CmlkLoYJVpBksNScEI1IAshzDd0 sdE6s+SVNfjLCrx2sG8aVVIfNYXWbud3fZ34yj5HnoRym18wl/MbXeL58ygJdWKC B7c4UL1hanu6OouM9win3n6QRUmeARAKtCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1660732929; x= 1660819329; bh=/e+pY4F2y5e07H0P8Rm+Cgz01iTckno/oZxX5csVqDc=; b=K e+E0j+XisUGzZ6Ol6xp8FtWVZkEUeddzSVsdcNJoikEw3S1UeDW4i9x12LYb8doz 8fW+jseQkO+vUhIKpgrqj6IGZULJCSAmv74RZSL1frboXcxY7ngObxArdQrRk4ox HctZvbux0t4ObcY7fKUwzzIW4otItySousQpSK34NtKtcZ+XTLu/TlLwsaZLeSqJ XTMSirBKi0Ijx/4b8ZvV5ONIWn3aenMk1nvWNOoilfw0VSB8cwdp8faaz9xSw96T ZjXmk9cS+UjhpSldFmtVvWXyHLJ69gsikJSVOEyULBOM7CudHt7EEAPz0h1sCbNA +Y6lQuJYvqwBUVltXphUA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehiedgfeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesghdtreertd dtjeenucfhrhhomheptehlhihsshgrucftohhsshcuoehhihesrghlhihsshgrrdhisheq necuggftrfgrthhtvghrnhepfeeltdfgieelkeekhfegtedvkeegteetvefgheehgeehve eiheehveejleeiffdvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghi lhhfrhhomhepqhihlhhishhssegvvhgvrdhqhihlihhsshdrnhgvth X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 17 Aug 2022 06:42:08 -0400 (EDT) Received: by eve.qyliss.net (Postfix, from userid 1000) id 1D305581; Wed, 17 Aug 2022 10:42:06 +0000 (UTC) Date: Wed, 17 Aug 2022 10:42:06 +0000 From: Alyssa Ross To: discuss@spectrum-os.org Subject: virtio-gpu in cloud-hypervisor: which path do we take? Message-ID: <20220817104206.7k5oi7f3o3kcfdoo@eve> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="byqlutzcprj5kvj3" Content-Disposition: inline Message-ID-Hash: 5IN6OQ5ZV4Y3XWHKI2MNAVSL6ZGE5ERF X-Message-ID-Hash: 5IN6OQ5ZV4Y3XWHKI2MNAVSL6ZGE5ERF X-MailFrom: qyliss@eve.qyliss.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-discuss.spectrum-os.org-0; header-match-discuss.spectrum-os.org-1; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.5 Precedence: list List-Id: General high-level discussion about Spectrum Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --byqlutzcprj5kvj3 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Recently I've been working on making it possible for us to use crosvm's implementation of virtio-gpu (which is necessary for multi-VM Wayland). The approach I was originally planning on was porting crosvm's vhost-user-gpu frontend to cloud-hypervisor. That would allow us to run the crosvm implementation of the device unmodified, with just a small amount of glue code in cloud-hypervisor. But then I discovered some things that made me decide to investigate other approaches: - crosvm does not implement the standard vhost-user-gpu protocol. It implements it its own special way. Perhaps ironically, the crosvm-specific way seems to be closer to how vhost-user works for other devices (like network and block), which should actually make it easier to port the frontend to cloud-hypervisor. But it also changes the potential to upstream that port to cloud-hypervisor from "a hard sell" to "not going to happen". So if I did that, it would commit us to carrying a cloud-hypervisor patch indefinitely. - There's an interesting new protocol called vfio-user, that would be really helpful to us in this situation. Whereas vhost-user requires the VMM to still have some basic per-device knowledge (the glue code I was planning to port), vfio-user operates at the PCI level, so the VMM only needs to know that the device is PCI. So if we could somehow provide a virtio-gpu device to cloud-hypervisor over vfio-user, cloud-hypervisor wouldn't need any GPU-specific code at all. Everything should just work without any changes to cloud-hypervisor, as it already implements a vfio-user client. - The next release of QEMU, 7.1.0, will include support for exporting any virtual device QEMU can provide over vfio-user. So with all this in mind, there are three ways we could try to proceed: 1. Port the crosvm-specific vhost-user-gpu frontend to cloud-hypervisor. 2. Make crosvm speak the standard version of vhost-user-gpu, then use QEMU to act as a bridge between the crosvm GPU device, and cloud-hypervisor, translating vhost-user to vfio-user, but not doing anything else. (So we're not using QEMU to run a VM, just to translate between these two protocols and handle the PCI stuff.) 3. Implement a vfio-user server in crosvm, so crosvm device backends can be used directly with cloud-hypervisor. 3 is the clear best option, because it doesn't require adding QEMU into the system, and it would be entirely upstreamable =E2=80=94 I spoke to a cr= osvm developer about it in their Matrix channel and they said they'd be interested in patches for it. But it's also quite complicated to implement, and beyond my ability, at least if I want results any time soon. 2 would probably be even more complicated as it would require coordinating between crosvm and QEMU to get them to agree on how the protocol should work. So if we want this working soon, 1 is the only feasible option, at least if it's me doing the work. But it means committing to carrying a cloud-hypervisor patch until somebody comes along to implement 3. It gives me bad vibes because it goes against the upstream-first approach I try to take with Spectrum development, and once timely package updates are something we have to take more seriously, the patch no longer applying would block any sort of automatic updates. So I'm posting this to solicit thoughts on what to do here. The ideal scenario is that we are able to find somebody else (with more VMM implementation experience than me) who is able to do 3, and then I would be totally comfortable with doing 1 as a stopgap until that can happen. Otherwise, I can either keep trying to chip away at doing it myself, however long that takes, or we'd have to just accept the consequences of having the patch indefinitely, and hope that Google or somebody else also finds themself wishing crosvm had a vfio-user server and implements it themself. Thoughts welcome. :) --byqlutzcprj5kvj3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEH9wgcxqlHM/ARR3h+dvtSFmyccAFAmL8xfUACgkQ+dvtSFmy ccD08g/+LSol3Pb8oveytJlPHD+grZ6EICLhnGCXpzP75+g+mV/r3S25j1qfDnR/ IY59W8KOmOwMBJNrQszcLa2m0vfkVUwySIQqty0V6n+HnN31tdRguOBY7S2mNHq/ QF49MLlH6zctiq1Wk00HDlDZyye/u++lKdcfLTWjCN1/GXUwGzRYM9ZajXDHmgPR mdjPWaY5C3V904E3ofEPocwjC/Jte4TE/T96QkjVannB44ANpoiwreXGr8UglZ16 4pQZjdBf50bTDH4MUs5Eg3R9CQ6FNBZUy+usLhQImk7K58EyXdhfhXm3NMt99boS 9TgVB9ifq+X5Rz8WP6ntc9bnVHuee5u6MycxUMzOyWQdhbMHN8W6alXE3n+Dm9kn zjb++kxphutw3UW82GXvcTgt2o7onm5OTEMQOq6eTt9+hHq98TFSwnNC+afUBqi+ 1Y+BaObuwJi1t6JbUNKaUvqK21nOC58jHS0dhsjZSEyEGWsAtRlDxmLnBneWeh76 eGZJvukXYtCz4IrbkjXC26SzsENF9J1VZmu4WT+nHC2M97kWHP/pIwuDjLYtLwBt nE2pTolxX201RuNele5d2M7xFuINyIGzR5APd2Mxh7iEKAQL8Ug17Sk5DKZia2uV d5c5uONSkCAW02UUfAQCzgIcj1bVotSOQ3cGcDw4nM+3h+vKpCY= =xQfD -----END PGP SIGNATURE----- --byqlutzcprj5kvj3--