[PATCH] vm: optional 16K-page guest kernel builds

[PATCH] vm: optional 16K-page guest kernel builds

[colbyt]

From: PolymerOS Coordinator <coordinator@polymeros.local>

KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
and net VM kernel builds; the default is unchanged. A 16K-page guest
Image built this way has booted to a full session under a 16K host here
since June without issues on my side.
---
Following up on the offer in my July 7 mail, as a standalone patch now
that the nixpkgs update brings Linux 7.1. The config delta is five
lines; the edit (an argument defaulting to today's behavior) is a
proposal — happy to rework it if you'd rather structure it differently,
e.g. hang it off a single shared helper instead of two arguments.

 spectrum/img/app/default.nix    | 7 +++++++
 spectrum/vm/sys/net/default.nix | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/img/app/default.nix b/img/app/default.nix
index 06dfc86c..6b839f0f 100644
--- a/img/app/default.nix
+++ b/img/app/default.nix
@@ -7,6 +7,7 @@ import ../../lib/call-package.nix (
 , cryptsetup
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , cacert, linux_latest
+, kernelPageSize ? null
 }:
 
 let
@@ -49,6 +50,12 @@ let
       VSOCKETS = yes;
       VIRTIO_VSOCKETS = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
diff --git a/vm/sys/net/default.nix b/vm/sys/net/default.nix
index 3ec34a7f..efd1c8e1 100644
--- a/vm/sys/net/default.nix
+++ b/vm/sys/net/default.nix
@@ -10,6 +10,7 @@ pkgsMusl.callPackage (
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , busybox, dbus, execline, iwd, kmod, linuxPackagesFor, linux_latest
 , mdevd, nftables, s6, s6-linux-init, spectrum-driver-tools, xdp-tools
+, kernelPageSize ? null
 }:
 
 let
@@ -52,6 +53,12 @@ let
       VIRTIO_CONSOLE = yes;
       VIRTIO_PCI = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
-- 
2.55.0

[PATCH v2] vm: optional 16K-page guest kernel builds

[colbyt]

From: PolymerOS Coordinator <coordinator@polymeros.local>

KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
and net VM kernel builds; the default is unchanged. A 16K-page guest
Image built this way has booted to a full session under a 16K host here
since June without issues on my side.
---
v2: no code change — drop a stray in-body From line that would have
misattributed authorship when applied.

Following up on the offer in my July 7 mail, as a standalone patch now
that the nixpkgs update brings Linux 7.1. The config delta is five
lines; the edit (an argument defaulting to today's behavior) is a
proposal — happy to rework it if you'd rather structure it differently,
e.g. hang it off a single shared helper instead of two arguments.

 spectrum/img/app/default.nix    | 7 +++++++
 spectrum/vm/sys/net/default.nix | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/img/app/default.nix b/img/app/default.nix
index 06dfc86c..6b839f0f 100644
--- a/img/app/default.nix
+++ b/img/app/default.nix
@@ -7,6 +7,7 @@ import ../../lib/call-package.nix (
 , cryptsetup
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , cacert, linux_latest
+, kernelPageSize ? null
 }:
 
 let
@@ -49,6 +50,12 @@ let
       VSOCKETS = yes;
       VIRTIO_VSOCKETS = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
diff --git a/vm/sys/net/default.nix b/vm/sys/net/default.nix
index 3ec34a7f..efd1c8e1 100644
--- a/vm/sys/net/default.nix
+++ b/vm/sys/net/default.nix
@@ -10,6 +10,7 @@ pkgsMusl.callPackage (
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , busybox, dbus, execline, iwd, kmod, linuxPackagesFor, linux_latest
 , mdevd, nftables, s6, s6-linux-init, spectrum-driver-tools, xdp-tools
+, kernelPageSize ? null
 }:
 
 let
@@ -52,6 +53,12 @@ let
       VIRTIO_CONSOLE = yes;
       VIRTIO_PCI = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
-- 
2.55.0

[PATCH v3] vm: optional 16K-page guest kernel builds

[colbyt]

KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
and net VM kernel builds; the default is unchanged. A 16K-page guest
Image built this way has booted to a full session under a 16K host here
since June without issues on my side.
---
v3/v2: no code change — drop a stray in-body From line that would have
misattributed authorship when applied.

Following up on the offer in my July 7 mail, as a standalone patch now
that the nixpkgs update brings Linux 7.1. The config delta is five
lines; the edit (an argument defaulting to today's behavior) is a
proposal — happy to rework it if you'd rather structure it differently,
e.g. hang it off a single shared helper instead of two arguments.

 spectrum/img/app/default.nix    | 7 +++++++
 spectrum/vm/sys/net/default.nix | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/img/app/default.nix b/img/app/default.nix
index 06dfc86c..6b839f0f 100644
--- a/img/app/default.nix
+++ b/img/app/default.nix
@@ -7,6 +7,7 @@ import ../../lib/call-package.nix (
 , cryptsetup
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , cacert, linux_latest
+, kernelPageSize ? null
 }:
 
 let
@@ -49,6 +50,12 @@ let
       VSOCKETS = yes;
       VIRTIO_VSOCKETS = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
diff --git a/vm/sys/net/default.nix b/vm/sys/net/default.nix
index 3ec34a7f..efd1c8e1 100644
--- a/vm/sys/net/default.nix
+++ b/vm/sys/net/default.nix
@@ -10,6 +10,7 @@ pkgsMusl.callPackage (
 , erofs-utils, jq, lndir, s6-rc, util-linux
 , busybox, dbus, execline, iwd, kmod, linuxPackagesFor, linux_latest
 , mdevd, nftables, s6, s6-linux-init, spectrum-driver-tools, xdp-tools
+, kernelPageSize ? null
 }:
 
 let
@@ -52,6 +53,12 @@ let
       VIRTIO_CONSOLE = yes;
       VIRTIO_PCI = yes;
       VT = no;
+    } // lib.optionalAttrs (kernelPageSize == "16k") {
+      # KVM hosts with a 16K stage-2 granule (e.g. Apple Silicon, NVIDIA
+      # GB10) can only run guests whose kernel uses 16K pages.
+      ARM64_4K_PAGES = lib.mkForce no;
+      ARM64_16K_PAGES = lib.mkForce yes;
+      ARM64_64K_PAGES = lib.mkForce no;
     };
   }).overrideAttrs ({ installFlags ? [], ... }: {
     installFlags = installFlags ++ [
-- 
2.55.0

Re: [PATCH v3] vm: optional 16K-page guest kernel builds

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]

colbyt <colby@colbyt.com> writes:

> KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
> upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
> with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
> and net VM kernel builds; the default is unchanged. A 16K-page guest
> Image built this way has booted to a full session under a 16K host here
> since June without issues on my side.
> ---
> v3/v2: no code change — drop a stray in-body From line that would have
> misattributed authorship when applied.
>
> Following up on the offer in my July 7 mail, as a standalone patch now
> that the nixpkgs update brings Linux 7.1. The config delta is five
> lines; the edit (an argument defaulting to today's behavior) is a
> proposal — happy to rework it if you'd rather structure it differently,
> e.g. hang it off a single shared helper instead of two arguments.

My instinct is that we should just always use a 16K page kernel on
aarch64 hosts.  AIUI the main reason most Linux distributions don't is
to retain compatibility with binaries that may assume only 4K alignment
is needed, but on the Spectrum host we should never be running binaries
we didn't build ourselves.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH v3] vm: optional 16K-page guest kernel builds

[colby]

Seems fine, I don't have any special perspective on that.  Everything I have is 16k.



On Friday, July 31st, 2026 at 7:04 AM, Alyssa Ross <hi@alyssa.is> wrote:

> colbyt <colby@colbyt.com> writes:
> 
> > KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
> > upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
> > with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
> > and net VM kernel builds; the default is unchanged. A 16K-page guest
> > Image built this way has booted to a full session under a 16K host here
> > since June without issues on my side.
> > ---
> > v3/v2: no code change — drop a stray in-body From line that would have
> > misattributed authorship when applied.
> >
> > Following up on the offer in my July 7 mail, as a standalone patch now
> > that the nixpkgs update brings Linux 7.1. The config delta is five
> > lines; the edit (an argument defaulting to today's behavior) is a
> > proposal — happy to rework it if you'd rather structure it differently,
> > e.g. hang it off a single shared helper instead of two arguments.
> 
> My instinct is that we should just always use a 16K page kernel on
> aarch64 hosts.  AIUI the main reason most Linux distributions don't is
> to retain compatibility with binaries that may assume only 4K alignment
> is needed, but on the Spectrum host we should never be running binaries
> we didn't build ourselves.
>

Re: [PATCH v3] vm: optional 16K-page guest kernel builds

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1665 bytes --]

On Sat, Aug 01, 2026 at 06:40:48PM +0000, colby@colbyt.com wrote:
> Seems fine, I don't have any special perspective on that.  Everything I have is 16k.

Alright, I'll throw that in next time we're doing a kernel rebuild. :)

> On Friday, July 31st, 2026 at 7:04 AM, Alyssa Ross <hi@alyssa.is> wrote:
>
> > colbyt <colby@colbyt.com> writes:
> >
>
> > > KVM hosts with a 16K stage-2 granule (Apple Silicon machines running
> > > upstream kernels, NVIDIA GB10) can only run guests whose kernel is built
> > > with ARM64_16K_PAGES. Add an optional kernelPageSize argument to the app
> > > and net VM kernel builds; the default is unchanged. A 16K-page guest
> > > Image built this way has booted to a full session under a 16K host here
> > > since June without issues on my side.
> > > ---
> > > v3/v2: no code change — drop a stray in-body From line that would have
> > > misattributed authorship when applied.
> > >
> > > Following up on the offer in my July 7 mail, as a standalone patch now
> > > that the nixpkgs update brings Linux 7.1. The config delta is five
> > > lines; the edit (an argument defaulting to today's behavior) is a
> > > proposal — happy to rework it if you'd rather structure it differently,
> > > e.g. hang it off a single shared helper instead of two arguments.
> >
>
> > My instinct is that we should just always use a 16K page kernel on
> > aarch64 hosts.  AIUI the main reason most Linux distributions don't is
> > to retain compatibility with binaries that may assume only 4K alignment
> > is needed, but on the Spectrum host we should never be running binaries
> > we didn't build ourselves.
> >


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]