Re: [PATCH] release/checks/try.nix: init

[Ryan Lahfa <ryan@lahfa.xyz>]

Reviewed-by: Ryan Lahfa <ryan@lahfa.xyz>

On Sat, May 27, 2023 at 03:07:20PM +0000, Alyssa Ross wrote:
> Ryan Lahfa <ryan@lahfa.xyz> writes:
> 
> > On Sat, May 27, 2023 at 02:38:22PM +0000, Alyssa Ross wrote:
> >> Ryan Lahfa <ryan@lahfa.xyz> writes:
> >> 
> >> > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote:
> >> >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext
> >> >> partition to fail").  It's the first time we're actually doing
> >> >> automated tests of a Spectrum boot.  For now, I'm using the NixOS test
> >> >> framework, but because we're not using NixOS and not setting any NixOS
> >> >> options, it feels to me like it doesn't actually buy us very much, so
> >> >> if it doesn't start adding more value as we add more (or more complex)
> >> >> tests, it might be simpler to just use a shell/execline script for
> >> >> tests.
> >> >
> >> > Are you only interested into tests that works without any
> >> > instrumentation?
> >> >
> >> > e.g. what if Spectrum added the backdoor.service in their service
> >> > management? That'd repair the ability of the test framework to have
> >> > better interactions with the guest without QEMU Agent.
> >> 
> >> At least until I really need tests that depend on guest cooperation, yes.
> >> Because to have either the backdoor service or the QEMU agent, I'd
> >> either have to build custom images just for testing (which would mean
> >> the real images were not actually tested), or I'd have to build those
> >> things into all Spectrum images.
> >> 
> >> At some point, it might not be possible to get away from this, but the
> >> basic tests I've written so far have all gone fine without the need for
> >> any guest cooperation beyond a serial shell.
> >
> > Makes sense, supporting "minimal" friction for this usecase is
> > desirable.
> >
> > I guess we could have Python *and* Bash/execline as "test scripts" if we want.
> >
> > But I would need to see more about what would Bash/execline-oriented
> > test scripts would look like in practice.
> >
> > Python is suboptimal for various things but the "batteries included" are
> > very useful for getting a OK-grade thing running.
> 
> I don't think there's much of a reason to support shell/execline scripts
> for NixOS tests.  My point here is more that I'm not really getting
> anything out of the NixOS test setup at all.  I already have to
> construct my own QEMU command line, so the only thing I'm really getting
> out of it is the wait_for_console_text() function, which wouldn't be
> _that_ hard to do without in a shell/execline script either.
> 
> It would probably be different if I _was_ using a guest agent / backdoor
> service, so I do think there's value to the NixOS test framework trying
> to support other distros.  It would be useful when the tests would
> benefit from its built-in systemd integration, or OCR functionality, etc.
> 
> >> >> +config.pkgs.nixosTest ({ pkgs, ... }: {
> >> >> +  name = "try-spectrum-test";
> >> >> +  nodes = {};
> >> >> +
> >> >> +  testScript = ''
> >> >> +    import shlex
> >> >> +    import subprocess
> >> >> +
> >> >> +    conf = subprocess.run([
> >> >> +      "${pkgs.mtools}/bin/mcopy",
> >> >> +      "-i",
> >> >> +      "${live}@@1M",
> >> >> +      "::loader/entries/spectrum.conf",
> >> >> +      "-",
> >> >> +    ], stdout=subprocess.PIPE)
> >> >> +    conf.check_returncode()
> >> >> +
> >> >> +    cmdline = None
> >> >> +    for line in conf.stdout.decode('utf-8').splitlines():
> >> >> +      key, value = line.split(' ', 1)
> >> >> +      if key == 'options':
> >> >> +        cmdline = value
> >> >
> >> > Is there any reason to not have `conf` / `cmdline`
> >> > being derived in a derivation?
> >> 
> >> We can't know it at eval time, because it includes the verity hash.  So
> >> our options are to recompute it here, or reuse the results of the
> >> derivation that already computed it, which is the approach I've taken
> >> here.  It's a bit unweildy, but I blame Python for that.  If we do end
> >> up switching to shell it'd just be:
> >> 
> >> 	mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p'
> >
> > Makes sense, but do you need it at evaltime?
> >
> > Wouldn't this be a possibility:
> >
> > ```nix
> > let optionsFile = pkgs.runCommand "extract-options" {} ''mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > $out'';
> > in
> > ''
> >   with open(${optionsFile}, "r") as f:
> >     options = f.read()
> > ''
> > ```
> >
> > Ideally, with the https://github.com/NixOS/nixpkgs/issues/156563
> > proposal, this could even become easier IMHO.
> 
> Well yeah, I could, but I think that's just extra complexity over doing
> it in the driver script.  It's not _that_ bad in Python.



-- 
Ryan Lahfa