Re: [PATCH] release/checks/try.nix: init

Alyssa Ross
2023-05-27T15:07:36Z

[-- Attachment #1: Type: text/plain, Size: 4561 bytes --]

Ryan Lahfa <ryan@lahfa.xyz> writes:

> On Sat, May 27, 2023 at 02:38:22PM +0000, Alyssa Ross wrote:
>> Ryan Lahfa <ryan@lahfa.xyz> writes:
>>
>> > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote:
>> >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext
>> >> partition to fail"). It's the first time we're actually doing
>> >> automated tests of a Spectrum boot. For now, I'm using the NixOS test
>> >> framework, but because we're not using NixOS and not setting any NixOS
>> >> options, it feels to me like it doesn't actually buy us very much, so
>> >> if it doesn't start adding more value as we add more (or more complex)
>> >> tests, it might be simpler to just use a shell/execline script for
>> >> tests.
>> >
>> > Are you only interested into tests that works without any
>> > instrumentation?
>> >
>> > e.g. what if Spectrum added the backdoor.service in their service
>> > management? That'd repair the ability of the test framework to have
>> > better interactions with the guest without QEMU Agent.
>>
>> At least until I really need tests that depend on guest cooperation, yes.
>> Because to have either the backdoor service or the QEMU agent, I'd
>> either have to build custom images just for testing (which would mean
>> the real images were not actually tested), or I'd have to build those
>> things into all Spectrum images.
>>
>> At some point, it might not be possible to get away from this, but the
>> basic tests I've written so far have all gone fine without the need for
>> any guest cooperation beyond a serial shell.
>
> Makes sense, supporting "minimal" friction for this usecase is
> desirable.
>
> I guess we could have Python *and* Bash/execline as "test scripts" if we want.
>
> But I would need to see more about what would Bash/execline-oriented
> test scripts would look like in practice.
>
> Python is suboptimal for various things but the "batteries included" are
> very useful for getting a OK-grade thing running.

I don't think there's much of a reason to support shell/execline scripts
for NixOS tests. My point here is more that I'm not really getting
anything out of the NixOS test setup at all. I already have to
construct my own QEMU command line, so the only thing I'm really getting
out of it is the wait_for_console_text() function, which wouldn't be
_that_ hard to do without in a shell/execline script either.

It would probably be different if I _was_ using a guest agent / backdoor
service, so I do think there's value to the NixOS test framework trying
to support other distros. It would be useful when the tests would
benefit from its built-in systemd integration, or OCR functionality, etc.

>> >> +config.pkgs.nixosTest ({ pkgs, ... }: {
>> >> + name = "try-spectrum-test";
>> >> + nodes = {};
>> >> +
>> >> + testScript = ''
>> >> + import shlex
>> >> + import subprocess
>> >> +
>> >> + conf = subprocess.run([
>> >> + "${pkgs.mtools}/bin/mcopy",
>> >> + "-i",
>> >> + "${live}@@1M",
>> >> + "::loader/entries/spectrum.conf",
>> >> + "-",
>> >> + ], stdout=subprocess.PIPE)
>> >> + conf.check_returncode()
>> >> +
>> >> + cmdline = None
>> >> + for line in conf.stdout.decode('utf-8').splitlines():
>> >> + key, value = line.split(' ', 1)
>> >> + if key == 'options':
>> >> + cmdline = value
>> >
>> > Is there any reason to not have `conf` / `cmdline`
>> > being derived in a derivation?
>>
>> We can't know it at eval time, because it includes the verity hash. So
>> our options are to recompute it here, or reuse the results of the
>> derivation that already computed it, which is the approach I've taken
>> here. It's a bit unweildy, but I blame Python for that. If we do end
>> up switching to shell it'd just be:
>>
>> mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p'
>
> Makes sense, but do you need it at evaltime?
>
> Wouldn't this be a possibility:
>
> ```nix
> let optionsFile = pkgs.runCommand "extract-options" {} ''mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > $out'';
> in
> ''
> with open(${optionsFile}, "r") as f:
> options = f.read()
> ''
> ```
>
> Ideally, with the https://github.com/NixOS/nixpkgs/issues/156563
> proposal, this could even become easier IMHO.

Well yeah, I could, but I think that's just extra complexity over doing
it in the driver script. It's not _that_ bad in Python.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]