GPU virtualization in Spectrum

[Alyssa Ross <hi@alyssa.is>]

[-- Attachment #1: Type: text/plain, Size: 16734 bytes --]

Table of Contents
─────────────────

1. Display controllers and renderers
2. Overview of GPU virtualization technologies
.. 1. Direct passthrough
.. 2. Hardware virtualization and paravirtualization
..... 1. Intel
..... 2. NVIDIA
..... 3. AMD
..... 4. ARM
.. 3. Virgil 3D
3. Seamless windowing
.. 1. virtio-gpu 2d
.. 2. virtio-wl
.. 3. virtio-gpu context types
4. Putting it together — GPUs in Spectrum VMs


This report documents the current state of the GPU virtualization
ecosystem as it relates to Spectrum.

GPU virtualization is a very controversial topic in the world of
compartmentalized operating systems.  Qubes doesn't support GPU
acceleration of anything but the GUI VM (the equivalent of Spectrum's
compositor VM), and (I think as a result of this) a way to provide GPU
accelerate to applications is *the* most requested Spectrum feature by a
large margin.

But there are serious security concerns with GPU virtualization.  GPUs
have a frankly awful track record when it comes to isolation.  This
might improve going forward, especially with SR-IOV-based solutions,
because business demand for secure GPU virtualization is increasing and
GPU vendors are trying to meet that demand.  For example, AMD says of
their GPU virtualization implementation[1]:

      The hardware-enforced memory isolation logic provides strong
      data security among the VFs, which helps prevent one VM from
      being able to access another VM’s data.

      With security being a bare minimum requirement for any
      virtualization solution, AMD’s hardware-based virtualized
      GPU solution offers a strong deterrent to unauthorized users
      who traverse the software or application layers seeking
      means to extract or corrupt GPU user data from the virtual
      machines.  Although a VF can access full GPU capabilities at
      its own GPU partition, it does not have access to the
      dedicated local memory of its sibling VFs.

But it remains to be seen whether these claims will stand up in
practice, especially because SR-IOV is not yet widely available.  The
alternatives (software multiplexing and GPU emulation) have both seen
exploitable security issues.

Because of this, my current plan is that access to the GPU will be
highly restricted in a default Spectrum system.  I do think, though,
that there needs to be a way for users to opt in to using whatever
virtualization features their hardware provides, because I've heard so
many accounts from people who find themselves unable to use
compartmentalized systems /at all/ because of certain tasks where they
need graphics acceleration.  My primary goal with Spectrum is to bring
compartmentalized computing to people who are not currently using it,
and to people who're currently using mainstream systems where any
application they run might be stealing their private SSH keys or
ransomwaring all their files, being potentially vulnerable to zero-days
in GPU drivers or hardware is the least of their problems.


1 Display controllers and renderers
═══════════════════════════════════

  Standalone GPUs, and Intel integrated GPUs, tend to be display
  controllers and renderers all in a single package.  The distinction
  between them will be important, though, so briefly:

  A *display controller* is the hardware that takes care of making a
  pixel grid show up on a screen.

  A *renderer* does computation (3D renderering, etc.) to create an
  image that can be given to a display controller to display.
  (Naturally in can also be used for other computations as well.)


2 Overview of GPU virtualization technologies
═════════════════════════════════════════════

2.1 Direct passthrough
──────────────────────

  Direct passthrough is the simplest and least interesting option for
  accelerated graphics in a VM.  The host simply gives the VM control
  over a whole GPU.  Since we're aiming for one VM per application
  instance in Spectrum, direct passthrough might be useful if you have a
  dual GPU system, and want to accelerate one particular application,
  like a game.  Of course, direct passthrough is also the most secure
  sort of GPU virtualization — data leaks are far less likely to happen
  when each GPU user has their own dedicated hardware!  We can
  definitely support this.


2.2 Hardware virtualization and paravirtualization
──────────────────────────────────────────────────

  By *hardware virtualization*, I mean that the virtualization is
  implemented by the GPU itself, and by *paravirtualization* I mean that
  the host kernel implements virtual GPUs by multiplexing.

  The standard way of doing hardware virtualization is called SR-IOV,
  and it's widely used today with server network cards, and to a lesser
  extent NVMe drives.  It's not widely available on GPUs, especially not
  consumer ones, but it seems like that's about to change.

  When using this sort of virtualization, all that's being virtualized
  is the renderer.  The display controller stays attached to the host.
  I'm not aware of any implementation for hardware that is both a
  display controller and a renderer that supports separately passing
  through a display controller to a VM[2].


2.2.1 Intel
╌╌╌╌╌╌╌╌╌╌╌

  Intel's GPU paravirtualization technology is called GVT-g.  Unlike
  solutions from other GPU companies, GVT-g is very widely available —
  it's supported for integrated graphics starting from Broadwell
  (launched 2014).  GVT-g is definitely something we can support in
  Spectrum, but one limitation I found is that on my laptop (a Google
  Pixelbook), I can create at most two vGPUs, so don't think you're
  going to be able to have every application using accelerated graphics
  using GVT-g even if you're okay with the security implications.

  The most recent Intel GPUs no longer support GVT-g — they have
  hardware SR-IOV implementations instead.  I don't believe this is
  supported in Linux yet, but I imagine it will be at some point.


2.2.2 NVIDIA
╌╌╌╌╌╌╌╌╌╌╌╌

  NVIDIA officially supports GPU virtualization for most of their
  datacentre and professional GPUs, through their proprietary driver.
  Because of NVIDIA's general allergy to publishing code or
  documentation, I don't know how this GPU virtualization is
  implemented, but my guess is that it's mostly in hardware, because
  their product briefs claim their cards support SR-IOV.  There also
  exists third-party software that can modify NVIDIA's software to
  support some consumer cards.

  The GPUs that support it tend to support between 16 and 32 virtual
  GPUs.

  Obviously, running a proprietary driver in the Spectrum host kernel
  would have /horrible/ security implications, and I will always
  strongly advise against it.  But, we probably could make it work if we
  wanted to.


2.2.3 AMD
╌╌╌╌╌╌╌╌╌

  There are five AMD GPUs that implement hardware (SR-IOV)
  virtualization.  The Radeon Pro V520 is "only available as a public
  cloud offering" and so isn't a product you can actually buy.  The
  Radeon Pro V340 is only supported by VMware, not AMD's open source
  Linux driver.  The AMD S7100X, S7150, and S7150 x2 are all supported
  with KVM, but none of them have been produced since 2016.  It would
  certainly be possible to support those last three in Spectrum, but due
  to just how obscure AMD GPU virtualization is, I don't see myself
  prioritizing it any time soon.


2.2.4 ARM
╌╌╌╌╌╌╌╌╌

  There's single ARM GPU that supports hardware virtualization, the
  Mali-G78AE.  It's intended for "automotive and industrial"
  applications, so I doubt it's going to show up in any hardware anybody
  wants to run Spectrum on.


2.3 Virgil 3D
─────────────

  Virgil 3D takes a different approach, by doing entirely software-based
  GPU virtualization.  It implements OpenGL in software, in a way that's
  still backed by the host system's hardware GPU.  This is naturally
  less performant than hardware virtualization or paravirtualization due
  to the extra overhead, but the advantages are that you can get a lot
  of virtual GPUs out of it, since every VM is just another application
  using the GPU on the host, and that this approach works even when
  there isn't special support for virtualizing a particular GPU.


3 Seamless windowing
════════════════════

  One of the earliest features that Spectrum committed to was a seamless
  windowing experience, where applications running in different VMs
  would appear as native windows in the same Wayland compositor.


3.1 virtio-gpu 2d
─────────────────

  The simplest way to do seamless windowing is to run a Wayland
  compositor in each VM, and have it render windows to a virtual 2D GPU,
  where they can be displayed by a host compositor.  To the host
  compositor, windows are basically just opaque rectangles.  This is
  effectively how WSLg, the Windows Subsystem for Linux GUI, works.

  This works okay, but you lose the ability to have any integration
  between the application windows and the window manager they're being
  displayed on.  It's not actually seamless.  That's why, in screenshots
  of WSLg, you see Weston window chrome, instead of Windows ones.  The
  WSLg developers can either choose to draw Windows window decorations
  around every application (which will look bad for GNOME programs that
  draw their own decorations), or none of them (which means they have to
  use Weston's decorations, or there'd be windows with no decorations),
  because they can't directly speak the client-side decoration
  negotiation protocol to the applications.  This particular use case
  doesn't matter for Spectrum, because CSD is incompatible with
  unspoofable window decorations, but the point is that if you're not
  letting applications and window managers speak Wayland to each other,
  you need to come up with ad-hoc protocols every time you /do/ want
  them to be able to cooperate.

  Fortunately, there's another approach to seamless windowing, that
  preserves Wayland as the method of communication between applications
  and window manager.


3.2 virtio-wl
─────────────

  The original plan for seamless windowing in Spectrum was to use
  virtio-wl, a technology from Chromium OS.  virtio-wl (plus Sommelier,
  a component that runs in guest userspace) presents a socket-like
  interface, over which Wayland clients running in VMs can send and
  receive arbitrary data, as well as file descriptors, to and from a
  Wayland compositor running on the host.

  Because implementing support for sending arbitrary file descriptors
  out of a guest would be impossible, only certain types needed by the
  Wayland protocol are supported — pipes and host-allocated shared
  memory.  This worked fine for Wayland, but because of how
  Wayland-specific it is, virtio-wl was never considered suitable for
  upstreaming.

  This was a big problem for Spectrum specifically, because in virtio-wl
  there is no way for guests to share memory with the host.  In
  virtio-wl's intended use case, the compositor runs on the host,
  clients run in guests, and the compositor handles all memory
  allocation, so memory only needs to ever be shared in one direction,
  host → guest.  But in Spectrum, we'd like to have the compositor in
  its own VM, so we'd need some extra mechanism to let the compositor VM
  allocate host memory[3].


3.3 virtio-gpu context types
────────────────────────────

  Google's second attempt at seamless VM Wayland windowing is a new
  mechanism called virtio-gpu context types.  Despite the "gpu" in the
  name, there's not (necessarily) any GPU involved — it's just
  convenient to use virtio-gpu as a transport since it already has
  primitives for sharing memory between guest and host.

  Context types allow new protocols, like Wayland or Vulkan, to be sent
  over virtio-gpu, in addition to the 2D and Virgil 3D protocols it was
  designed for.  Context types should be available in Linux from v5.15
  (the next release at the time of writing).  Support for Wayland over
  virtio-gpu has already been implemented in crosvm and Sommelier.

  Even more excitingly, virtio-gpu has a mechanism by which guest
  userspace can do a special memory allocation that can be shared with
  the host!  So we won't need to add any special memory allocation
  mechanism just for Spectrum — we'll just need to make sure the
  compositor VM userspace knows how to allocate memory and send it to
  the host in this way.


4 Putting it together — GPUs in Spectrum VMs
═════════════════════════════════════════════

  After extensively researching all this, my current plan for graphics
  in Spectrum is as follows: each VM that runs a graphical application
  will get a virtio-gpu device, over which it will be expected to speak
  Wayland to the compositor.  VMs with accelerated graphics (which will
  be opt-in per-VM) will additionally get another GPU device to use for
  rendering — this could be, for example, a GVT-g vGPU, or it could be a
  Virgil 3D virtio-gpu.  They can use that accelerator (or software
  rendering if they don't have one) to render window contents into a
  buffer that can then be shared with the host (and by extension the
  compositor VM) by way of Wayland over virtio-gpu.  Rendering on a GPU
  while using Wayland over virtio-gpu isn't currently supported by
  Sommelier, but Google has it planned[4].

  A further problem we need to solve is how the compositor renders to
  the display controller.  If we can pass the display controller through
  to the compositor VM (this might be a possibility on some ARM
  systems), all's good.  But otherwise, we'd need to render to a virtual
  2D GPU, and then have the host display that on the actual display
  controller.  This is something that could probably be implemented
  fairly easily in crosvm's virtio-gpu stack — it already supports
  showing output as Wayland or X11 windows.  And in the meantime, we
  could cheat by running the world's simplest Wayland compositor on the
  host, which would just show the crosvm window and take care of talking
  to the display controller.

  Graphics and windowing is definitely something that we'll be iterating
  on for a very long time.  We can get started with the happy path
  Google already has working, with the compositor on the host.  From
  there, we can move the compositor into a guest, with direct GPU
  passthrough.  And then we can start looking at GPU virtualization.
  This field is evolving /extremely/ quickly, so by the time we get
  there things will undoubtedly have moved on.  That means it's very
  important not to work too much on it too early, because that work has
  a very high chance of being obsoleted shortly afterwards.  The purpose
  of this report is to demonstrate to the community that GPU accelerated
  applications in Spectrum will be possible by presenting how they would
  be implemented if I were to do so today.  But in reality, it makes
  much more sense to focus on almost everything else first, because no
  other area of virtualization is moving as fast as this one.



Footnotes
─────────

[1]
<https://www.amd.com/system/files/documents/amd-mxgpu-white-paper.pdf>

[2] (This would be direct passthrough, because if you have a graphics
card with one DisplayPort output, there's physically not much you can
do to share that output between multiple VMs.)

[3] I actually implemented this, before Wayland-over-virtio-gpu
emerged as a viable alternative that didn't need an extra mechanism
for this:
<https://spectrum-os.org/git/crosvm/tree/servers/src/memfd.rs?h=interguest&id=e5db778607e330a169e81f6b1d03648e0afaac6c>

[4]
<https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592#note_867838>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]