Ryan Lahfa <ryan@lahfa.xyz> writes:

> On Sat, May 27, 2023 at 02:38:22PM +0000, Alyssa Ross wrote:
>> Ryan Lahfa <ryan@lahfa.xyz> writes:
>> 
>> > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote:
>> >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext
>> >> partition to fail").  It's the first time we're actually doing
>> >> automated tests of a Spectrum boot.  For now, I'm using the NixOS test
>> >> framework, but because we're not using NixOS and not setting any NixOS
>> >> options, it feels to me like it doesn't actually buy us very much, so
>> >> if it doesn't start adding more value as we add more (or more complex)
>> >> tests, it might be simpler to just use a shell/execline script for
>> >> tests.
>> >
>> > Are you only interested into tests that works without any
>> > instrumentation?
>> >
>> > e.g. what if Spectrum added the backdoor.service in their service
>> > management? That'd repair the ability of the test framework to have
>> > better interactions with the guest without QEMU Agent.
>> 
>> At least until I really need tests that depend on guest cooperation, yes.
>> Because to have either the backdoor service or the QEMU agent, I'd
>> either have to build custom images just for testing (which would mean
>> the real images were not actually tested), or I'd have to build those
>> things into all Spectrum images.
>> 
>> At some point, it might not be possible to get away from this, but the
>> basic tests I've written so far have all gone fine without the need for
>> any guest cooperation beyond a serial shell.
>
> Makes sense, supporting "minimal" friction for this usecase is
> desirable.
>
> I guess we could have Python *and* Bash/execline as "test scripts" if we want.
>
> But I would need to see more about what would Bash/execline-oriented
> test scripts would look like in practice.
>
> Python is suboptimal for various things but the "batteries included" are
> very useful for getting a OK-grade thing running.

I don't think there's much of a reason to support shell/execline scripts
for NixOS tests.  My point here is more that I'm not really getting
anything out of the NixOS test setup at all.  I already have to
construct my own QEMU command line, so the only thing I'm really getting
out of it is the wait_for_console_text() function, which wouldn't be
_that_ hard to do without in a shell/execline script either.

It would probably be different if I _was_ using a guest agent / backdoor
service, so I do think there's value to the NixOS test framework trying
to support other distros.  It would be useful when the tests would
benefit from its built-in systemd integration, or OCR functionality, etc.

>> >> +config.pkgs.nixosTest ({ pkgs, ... }: {
>> >> +  name = "try-spectrum-test";
>> >> +  nodes = {};
>> >> +
>> >> +  testScript = ''
>> >> +    import shlex
>> >> +    import subprocess
>> >> +
>> >> +    conf = subprocess.run([
>> >> +      "${pkgs.mtools}/bin/mcopy",
>> >> +      "-i",
>> >> +      "${live}@@1M",
>> >> +      "::loader/entries/spectrum.conf",
>> >> +      "-",
>> >> +    ], stdout=subprocess.PIPE)
>> >> +    conf.check_returncode()
>> >> +
>> >> +    cmdline = None
>> >> +    for line in conf.stdout.decode('utf-8').splitlines():
>> >> +      key, value = line.split(' ', 1)
>> >> +      if key == 'options':
>> >> +        cmdline = value
>> >
>> > Is there any reason to not have `conf` / `cmdline`
>> > being derived in a derivation?
>> 
>> We can't know it at eval time, because it includes the verity hash.  So
>> our options are to recompute it here, or reuse the results of the
>> derivation that already computed it, which is the approach I've taken
>> here.  It's a bit unweildy, but I blame Python for that.  If we do end
>> up switching to shell it'd just be:
>> 
>> 	mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p'
>
> Makes sense, but do you need it at evaltime?
>
> Wouldn't this be a possibility:
>
> ```nix
> let optionsFile = pkgs.runCommand "extract-options" {} ''mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > $out'';
> in
> ''
>   with open(${optionsFile}, "r") as f:
>     options = f.read()
> ''
> ```
>
> Ideally, with the https://github.com/NixOS/nixpkgs/issues/156563
> proposal, this could even become easier IMHO.

Well yeah, I could, but I think that's just extra complexity over doing
it in the driver script.  It's not _that_ bad in Python.