From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 602A0BC646; Sat, 27 May 2023 15:09:57 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id 8EE89BC63F; Sat, 27 May 2023 15:09:54 +0000 (UTC) Received: from kurisu.lahfa.xyz (unknown [IPv6:2001:bc8:38ee::1]) by atuin.qyliss.net (Postfix) with ESMTPS id 69F83BC6B5 for ; Sat, 27 May 2023 15:09:50 +0000 (UTC) Date: Sat, 27 May 2023 17:09:39 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lahfa.xyz; s=kurisu; t=1685200181; bh=FMCDJAvO3SrEwqEb8dycNHpp7Oy9F8lXGzg+NBYfaWg=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=O1HIiCP7k9wfwklG34UR3F2z+nTjLsg2YKdMYwXiNsHrYeTsbu5+0nDC7uC4GuSrB fpdqUHSMhU4OEt9Nt1YonIKQ8/8bR9DH3CvpKwQQN1mCyb4x1/lSH/PFZhQdbyp2qE 4cvyIrVAPUKS/1scIFiCyoSarCQjS1rI5kfKIjzA= From: Ryan Lahfa To: Alyssa Ross Subject: Re: [PATCH] release/checks/try.nix: init Message-ID: References: <20230526210757.397735-1-hi@alyssa.is> <87353halgh.fsf@alyssa.is> <87y1l995jr.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87y1l995jr.fsf@alyssa.is> Message-ID-Hash: D4PILY4E2ZI46YRM3XBOPAE6HZX5JSRR X-Message-ID-Hash: D4PILY4E2ZI46YRM3XBOPAE6HZX5JSRR X-MailFrom: ryan@lahfa.xyz X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: devel@spectrum-os.org X-Mailman-Version: 3.3.5 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Sat, May 27, 2023 at 03:07:20PM +0000, Alyssa Ross wrote: > Ryan Lahfa writes: > > > On Sat, May 27, 2023 at 02:38:22PM +0000, Alyssa Ross wrote: > >> Ryan Lahfa writes: > >> > >> > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote: > >> >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext > >> >> partition to fail"). It's the first time we're actually doing > >> >> automated tests of a Spectrum boot. For now, I'm using the NixOS test > >> >> framework, but because we're not using NixOS and not setting any NixOS > >> >> options, it feels to me like it doesn't actually buy us very much, so > >> >> if it doesn't start adding more value as we add more (or more complex) > >> >> tests, it might be simpler to just use a shell/execline script for > >> >> tests. > >> > > >> > Are you only interested into tests that works without any > >> > instrumentation? > >> > > >> > e.g. what if Spectrum added the backdoor.service in their service > >> > management? That'd repair the ability of the test framework to have > >> > better interactions with the guest without QEMU Agent. > >> > >> At least until I really need tests that depend on guest cooperation, yes. > >> Because to have either the backdoor service or the QEMU agent, I'd > >> either have to build custom images just for testing (which would mean > >> the real images were not actually tested), or I'd have to build those > >> things into all Spectrum images. > >> > >> At some point, it might not be possible to get away from this, but the > >> basic tests I've written so far have all gone fine without the need for > >> any guest cooperation beyond a serial shell. > > > > Makes sense, supporting "minimal" friction for this usecase is > > desirable. > > > > I guess we could have Python *and* Bash/execline as "test scripts" if we want. > > > > But I would need to see more about what would Bash/execline-oriented > > test scripts would look like in practice. > > > > Python is suboptimal for various things but the "batteries included" are > > very useful for getting a OK-grade thing running. > > I don't think there's much of a reason to support shell/execline scripts > for NixOS tests. My point here is more that I'm not really getting > anything out of the NixOS test setup at all. I already have to > construct my own QEMU command line, so the only thing I'm really getting > out of it is the wait_for_console_text() function, which wouldn't be > _that_ hard to do without in a shell/execline script either. > > It would probably be different if I _was_ using a guest agent / backdoor > service, so I do think there's value to the NixOS test framework trying > to support other distros. It would be useful when the tests would > benefit from its built-in systemd integration, or OCR functionality, etc. Got it. > >> >> +config.pkgs.nixosTest ({ pkgs, ... }: { > >> >> + name = "try-spectrum-test"; > >> >> + nodes = {}; > >> >> + > >> >> + testScript = '' > >> >> + import shlex > >> >> + import subprocess > >> >> + > >> >> + conf = subprocess.run([ > >> >> + "${pkgs.mtools}/bin/mcopy", > >> >> + "-i", > >> >> + "${live}@@1M", > >> >> + "::loader/entries/spectrum.conf", > >> >> + "-", > >> >> + ], stdout=subprocess.PIPE) > >> >> + conf.check_returncode() > >> >> + > >> >> + cmdline = None > >> >> + for line in conf.stdout.decode('utf-8').splitlines(): > >> >> + key, value = line.split(' ', 1) > >> >> + if key == 'options': > >> >> + cmdline = value > >> > > >> > Is there any reason to not have `conf` / `cmdline` > >> > being derived in a derivation? > >> > >> We can't know it at eval time, because it includes the verity hash. So > >> our options are to recompute it here, or reuse the results of the > >> derivation that already computed it, which is the approach I've taken > >> here. It's a bit unweildy, but I blame Python for that. If we do end > >> up switching to shell it'd just be: > >> > >> mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > > > > Makes sense, but do you need it at evaltime? > > > > Wouldn't this be a possibility: > > > > ```nix > > let optionsFile = pkgs.runCommand "extract-options" {} ''mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > $out''; > > in > > '' > > with open(${optionsFile}, "r") as f: > > options = f.read() > > '' > > ``` > > > > Ideally, with the https://github.com/NixOS/nixpkgs/issues/156563 > > proposal, this could even become easier IMHO. > > Well yeah, I could, but I think that's just extra complexity over doing > it in the driver script. It's not _that_ bad in Python. Got it :). Kind regards, -- Ryan Lahfa