From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 180B0BC2EE; Sat, 27 May 2023 14:38:37 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id C344DBC2D9; Sat, 27 May 2023 14:38:31 +0000 (UTC) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by atuin.qyliss.net (Postfix) with ESMTPS id 3A3AFBC2D8 for ; Sat, 27 May 2023 14:38:28 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id B9E7A5C00F9; Sat, 27 May 2023 10:38:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Sat, 27 May 2023 10:38:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1685198306; x=1685284706; bh=BM zPaFQRgN+nVC2uG7y6xdwXt7jpeFl+M43izSe9ESU=; b=mkId4ZVbJkBEuFfekx OQylQOFpo1wUB7l3aRgtyMs/a7M6HzDP/odQQjS1NTavGv170yIOjcqumR/eu2zz H/ISkKfVvMmZoSdYkXw6aYQ6i8KEE5hX5X0DKY76KrB22e7BvAVd6D4fglNSlwrn laaGkATGCqLC7tSqPMky0yagKcH1U5D16/8J0PpDjsgdXrP9AZbIDM+VVl87KXSR EGDTut6myTEUDWp58spqj5hl9GjkjBQSWpgqiAReAocsXGRIiP6RtyKLseTdPwPk 5snyqogsgsvwmNpzscI0dlRRJ7ithRa3X5/Kx604wkKUZaHDOfLsBpptpWbQI8b9 fsDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1685198306; x=1685284706; bh=BMzPaFQRgN+nV C2uG7y6xdwXt7jpeFl+M43izSe9ESU=; b=vYgfMEjRek8ojM6Kl3GO9kwjcC2Bi kzbPflZsa+UbPGJDED/dXhEHPxe3YCF0KNRBKr/3Bf6Y0nt6jtjcHDuHUZTVSKkP 2tnY1s8CoWSbx13LI8JqWRoPO812aCvkCAmFnsORYJefzjeafjT7+jTVXnDjDrWC uHNN7AeAFoGQ17c4QUSuOzfYt3KUfOzJizbSFVkB8Yv04MS1Su1HiAxOqSAJXvQY sZ7IWNgz+DbosP1yYqxKMf95JOUmjG9rLRPcnyTfGKVEAsd1DUUkPEREWZ1NBQ6l RdyXQlKY1jwfb4sQBPS94AWvXiGsChJ8zWLa6MtNox/6Qts3AruVADY3g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeekuddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdludejmdenucfjughrpefhvfevufgjfhffkfggtgesghdtreertddt tdenucfhrhhomheptehlhihsshgrucftohhsshcuoehhihesrghlhihsshgrrdhisheqne cuggftrfgrthhtvghrnhepudetteduheekvdeutdevkeekteevgeejvdefvdejtefgueej ueetleeiuefftefhnecuffhomhgrihhnpehnihigohhsrdhorhhgpdhlihhnuhigqdhkvg hrnhgvlhdrthgrrhhgvghtpdhgihhthhhusgdrtghomhenucevlhhushhtvghrufhiiigv pedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehhihesrghlhihsshgrrdhish X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 27 May 2023 10:38:26 -0400 (EDT) Received: by x220.qyliss.net (Postfix, from userid 1000) id BC4F054B0; Sat, 27 May 2023 14:38:23 +0000 (UTC) From: Alyssa Ross To: Ryan Lahfa Subject: Re: [PATCH] release/checks/try.nix: init In-Reply-To: References: <20230526210757.397735-1-hi@alyssa.is> Date: Sat, 27 May 2023 14:38:22 +0000 Message-ID: <87353halgh.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Message-ID-Hash: E5Y3UHXM3SGDUAKSIDOYZT6GFEUTVYFF X-Message-ID-Hash: E5Y3UHXM3SGDUAKSIDOYZT6GFEUTVYFF X-MailFrom: hi@alyssa.is X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: devel@spectrum-os.org X-Mailman-Version: 3.3.5 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --=-=-= Content-Type: text/plain Ryan Lahfa writes: > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote: >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext >> partition to fail"). It's the first time we're actually doing >> automated tests of a Spectrum boot. For now, I'm using the NixOS test >> framework, but because we're not using NixOS and not setting any NixOS >> options, it feels to me like it doesn't actually buy us very much, so >> if it doesn't start adding more value as we add more (or more complex) >> tests, it might be simpler to just use a shell/execline script for >> tests. > > Are you only interested into tests that works without any > instrumentation? > > e.g. what if Spectrum added the backdoor.service in their service > management? That'd repair the ability of the test framework to have > better interactions with the guest without QEMU Agent. At least until I really need tests that depend on guest cooperation, yes. Because to have either the backdoor service or the QEMU agent, I'd either have to build custom images just for testing (which would mean the real images were not actually tested), or I'd have to build those things into all Spectrum images. At some point, it might not be possible to get away from this, but the basic tests I've written so far have all gone fine without the need for any guest cooperation beyond a serial shell. > On my side, I am interested supporting testing things like Debian + Nix > via the test framework and saying "here's a disk image or something, > boot it and connect to it". You might be interested in replying to this Discourse topic: https://discourse.nixos.org/t/running-a-nixos-test-on-a-different-image/28458 >> +config.pkgs.nixosTest ({ pkgs, ... }: { >> + name = "try-spectrum-test"; >> + nodes = {}; >> + >> + testScript = '' >> + import shlex >> + import subprocess >> + >> + conf = subprocess.run([ >> + "${pkgs.mtools}/bin/mcopy", >> + "-i", >> + "${live}@@1M", >> + "::loader/entries/spectrum.conf", >> + "-", >> + ], stdout=subprocess.PIPE) >> + conf.check_returncode() >> + >> + cmdline = None >> + for line in conf.stdout.decode('utf-8').splitlines(): >> + key, value = line.split(' ', 1) >> + if key == 'options': >> + cmdline = value > > Is there any reason to not have `conf` / `cmdline` > being derived in a derivation? We can't know it at eval time, because it includes the verity hash. So our options are to recompute it here, or reuse the results of the derivation that already computed it, which is the approach I've taken here. It's a bit unweildy, but I blame Python for that. If we do end up switching to shell it'd just be: mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' >> + >> + flags = " ".join(map(shlex.quote, [ >> + "qemu-kvm", >> + "-m", "512", >> + "-kernel", "${live.rootfs.kernel}/${pkgs.stdenv.hostPlatform.linux-kernel.target}", >> + "-initrd", "${live.initramfs}", >> + "-device", "qemu-xhci", >> + "-device", "usb-storage,drive=drive1,removable=true", >> + "-drive", "file=${live},id=drive1,format=raw,if=none,readonly=on", > > Does it bring anything that this is real "USB" device? (testing that the > drivers for USB are present?). > > In that case, shouldn't testing covers also NVMe devices, etc. ? USB is special in that USB devices don't show up until after userspace has started, so userspace has to handle waiting for them. This is not generally the case with other block devices. When working on the installer, I always test with USB, both to ensure this extra complexity is handled correctly, and because it's what I expect to be the most common device type for the combined installer image to be booted from in real life. >> + "-append", f"panic=-1 {cmdline}", >> + ])) >> + >> + machine = create_machine({"startCommand": flags}) > > `create_machine` is deprecated now, `Machine.create_startcommand` is > "deprecated" too, but I'm not really happy with the fact we do not have > serious replacements to that and I also want to better support "testing > non-NixOS" or non-instrumented systems in the framework. There's no viable alternative, so I think the best course of action here is to undeprecated it: https://github.com/NixOS/nixpkgs/pull/234427 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEH9wgcxqlHM/ARR3h+dvtSFmyccAFAmRyFd4ACgkQ+dvtSFmy ccA09BAAnBLp8ZcL4RlS+WqdHTmQfnhV1wvpWhSFmQ68oiiSZzkhHboR0cARPOOG piNzsCgvXDmS6kip7Z2zrI3MvD/UChD0yPLgtkx5KRv7tKqt/cXKNuug85YF2U/d k86XDIrgeh2nSehsIlQq6nuilc0xEPWqsdX4GKdIZ7EFKE97TRYGrtr1HzqpIux+ JhzPn8VAPQxj/BpWFPEy/xA7otxrUdSr2B07x91Xz2hb2TQGqI3LvtCivBibFpx1 S7FFt0VbhowBv5ElMSYjB8JH+LxjBWqMV3DHk3Q1L7Iwqprd0V4udxpahmbOhqev LgVBnuWqqXYIo1Cuu0EaaPCUbg5LstCk4ky0YGh0vkiTrkT7Bk/u5waR9cj43c8k keWEKpvZPClY0NvzSC7PAiLQlEdrvtqo9o0X8N9KsR3gdALGUQuklYYaDr1LUsEU Brpa8hCRpgfAK6rUuMnrnUItXeT6GJDIsDchFXPl7kCQDP7S6iF0m2rETkh0FqVa s1xeMN2IOndaZiu4ksrM/y1Uc4kE8TXNMLjh9gdHNUnWv8vnFOUpeqT68xhRbb66 Xp1T87IlBKWi9DdY592R9yxJoWga0u+b66EUVqj0wOeFkVar4vBgYmfoy0UDb0wP 6YVmtRAYih/4QEJRFwqCIGwnhy8HYguDJVfkmpO+X+3AGgFnTF4= =rjyy -----END PGP SIGNATURE----- --=-=-=--