Re: Using virtio-gpu instead of virtwl

Thomas Leonard
2022-03-21T12:11:36Z

On Wed, 16 Mar 2022 at 16:18, Thomas Leonard <talex5@gmail.com> wrote:
>
> On Tue, 15 Mar 2022 at 20:23, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
> > > On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
> [ virtio-gpu ]
> > > > I'm happy to test it now with Linux 5.16, but I think I need a newer
> > > > crosvm too. Looks like support got added in
> > > > https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
> > > > so it should be in 94.14150.
> > > >
> > > > Is there a new-enough package for crosvm available somewhere? NixOS
> > > > only has 81.12871.0.0-rc1, and spectrum seems to be on
> > > > 91.13904.0.0-rc2.
> > >
> > > I don't know of one. I haven't been keeping it up to date since we've
> > > mostly switched over to cloud-hypervisor for Spectrum, but it doesn't
> > > have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu
> > > implementation is on my list).
> > >
> > > I'll have a go at updating it in Nixpkgs and report back.
> >
> > Okay, here's a package for crosvm 99.14468.0.0-rc1:
> > https://github.com/NixOS/nixpkgs/pull/164306
>
> Thanks - that's very useful!
>
> Testing it from NixOS stable, it crashed with
[...]
> [src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to
> create an executor: Uring(CreatingContext(Setup(12)))',
> devices/src/virtio/block/asynchronous.rs:767:50
>
> From a brief look at the code, it looks like crosvm sees that the host
> is running Linux >= 5.10 and creates a test uring, which works. Then
> it tries to create a real one, and that fails. I fixed it by
> increasing the limit on locked memory in configuration.nix

However, this led to another problem: after resuming the host from
suspend, the crosvm drivers would crash with e.g.

[devices/src/virtio/block/asynchronous.rs:792] An error with a uring
source: URing::enter: Failed to enter io uring: 4

So a better fix is to run with "ulimit -l 0". That causes crosvm to
decide that uring isn't available and makes it fall back to the old
(working) system.

> > I've tested some basic stuff, but haven't tested virtio-gpu. I don't
> > see any reason it wouldn't work, though.
>
> I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`,
> crosvm crashed with:
>
> [ 31.326763] Invalid ELF header magic: != ELF
> [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0
> [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000
> [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible
> [ 31.333984] [drm] features: +context_init
> [ 31.337289] [drm] number of scanouts: 1
> [ 31.337938] [drm] number of cap sets: 1
> [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo
> 17, status 11, code 2
> [ERROR:devices/src/proxy.rs:212] failed write to child device process
> pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
>
> Then I tried with `--gpu=backend=2d` and that didn't crash, but
> instead opened a window showing some bootloader stuff. I now have
> /dev/dri/{card0, renderD128} devices, so I guess the next step is
> figuring out what they do!
>
> > Good luck! Let me know if I can help with anything else.

Looking at the Linux virtio_gpu driver, it seems that using contexts
requires virgl:

static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
*data, struct drm_file *file)
{
...
if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
return -EINVAL;

https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732

I think perhaps that crosvm is compiled without the "virgl_renderer"
feature (it's not in the default set), and this is causing it to crash
because that's also "self.default_component". I don't know how to
compile crosvm with virgl enabled, though.

--
talex5 (GitHub/Twitter)