From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 4CBA64392; Sat, 06 Dec 2025 15:29:32 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 993) id E78F8437F; Sat, 06 Dec 2025 15:29:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_PASS,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=4.0.1 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) by atuin.qyliss.net (Postfix) with ESMTPS id 6C489437D for ; Sat, 06 Dec 2025 15:29:28 +0000 (UTC) Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 997AB7A00B2; Sat, 6 Dec 2025 10:29:24 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Sat, 06 Dec 2025 10:29:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1765034964; x=1765121364; bh=qyw08yBrQA 7AbVnnSMbZvvFdrPb9g1PozCKXHQGOWbo=; b=tkQG2YFp2vKrLmfglPVYKbYbmQ WI0jKk8RW/Zti36AbkmWpyoX2ohkxH/7wr+qAIihxzQnfyU+kjVEou0lAur62Iz5 WWB4BdG/o5NpDuVpfnt0SjSs6aC8BJyBfj6llnDLQ/d9DucQ+LJRR1syeo3fxi45 bB1L5hXBzjB5t532hOihzzEuW6YD8QSIlriEcAKNwhHiF2ZPBvyS3B7a3SjJUIFd JekENYjYTfYRwiPY/amFpzewznvmUcmWH45clVKplz2Rz/M8L4pUJ/AOBG21TSwN +eNs13jB2mia12n04mAaIu8BwLJD5LA7PTSV0Z3NLVtlDpsHbOkSzcKE81pA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1765034964; x= 1765121364; bh=qyw08yBrQA7AbVnnSMbZvvFdrPb9g1PozCKXHQGOWbo=; b=l lYe5TfjErZ59iRG79olwRdo+LSp5nhcefdkSucJJT8x6FGimqt6w/bdYeiF0lLjV eVyrAhFjXlZAetys6d27ay8Lku+lZD+cGKzpOJ87kuaQmgBIk3guix8gx2JUDagw v2d7Kdnh/zCUzFc/kxbVC9dH3fCnkS/bhshcY9YMToWIrPT6LjH0A41B36XaPc69 miqAMoTTLr8YDxuAmdeYOiYaK1epjm109SwVSMn+lrULzeBhrpdEHpAy/c7Izp4l kO7gZuRiVsqe+nSiBej4cpLZ97q7GOYI58qrbxAv6O8ja3ors6rwxgH6cEiqnuRP BKDbpAUMXTjCeXBa/1ZFA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefgedrtddtgdduudeftdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpeffhffvvefukfhfgggtgfgjsehtkeertddttdejnecuhfhrohhmpeforghrvghkucfo rghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvhhish hisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrghtthgvrhhnpeeufeeiudet veekhffgleegheefkefhudevkeejuefhgedugeeigeegheevjeehjeenucffohhmrghinh epqhhusggvshdqohhsrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslhgvthhhihhnghhslh grsgdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtohepuggvmhhiohgsvghnohhurhesghhmrghilhdrtghomhdprhgtphhtthhopehhih esrghlhihsshgrrdhishdprhgtphhtthhopeguvghvvghlsehsphgvtghtrhhumhdqohhs rdhorhhgpdhrtghpthhtohepqhhusggvshdquggvvhgvlhesghhoohhglhgvghhrohhuph hsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 6 Dec 2025 10:29:23 -0500 (EST) Date: Sat, 6 Dec 2025 16:29:21 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Demi Marie Obenour Subject: Re: [qubes-devel] openQA bisectability Message-ID: References: <87bjkblrt0.fsf@alyssa.is> <21226c74-35a0-4bb3-b8f3-1163d4f5ad10@gmail.com> <878qfflrjm.fsf@alyssa.is> <875xajlojr.fsf@alyssa.is> <9e46e5d0-b2a7-4753-afcd-61ea74d1ddf9@gmail.com> <87345nlmf2.fsf@alyssa.is> <5017a410-0cee-4f15-8eed-dfd836cd42a0@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; x-action=pgp-signed Content-Transfer-Encoding: 8bit In-Reply-To: <5017a410-0cee-4f15-8eed-dfd836cd42a0@gmail.com> Message-ID-Hash: 6CNYBF23FZQDJYEO2IT7GWMFD3NBZU5V X-Message-ID-Hash: 6CNYBF23FZQDJYEO2IT7GWMFD3NBZU5V X-MailFrom: marmarek@invisiblethingslab.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; header-match-devel.spectrum-os.org-3; header-match-devel.spectrum-os.org-4; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Alyssa Ross , Spectrum OS Development , Qubes Developer Mailing List X-Mailman-Version: 3.3.9 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Sat, Dec 06, 2025 at 09:47:51AM -0500, Demi Marie Obenour wrote: > On 12/6/25 09:22, Alyssa Ross wrote: > > Demi Marie Obenour writes: > > > >> On 12/6/25 08:36, Alyssa Ross wrote: > >>> Demi Marie Obenour writes: > >>> > >>>> On 12/6/25 07:32, Alyssa Ross wrote: > >>>>> Demi Marie Obenour writes: > >>>>> > >>>>>> On 12/6/25 07:26, Alyssa Ross wrote: > >>>>>>> Demi Marie Obenour writes: > >>>>>>> > >>>>>>>> While trying to sandbox the file chooser portal, I broke it. > >>>>>>>> This caused files not to be saved, resulting in silent data loss. > >>>>>>>> Unfortunately, the integration test still passed. > >>>>>>>> > >>>>>>>> Is this a bug in the test? Is there a better alternative to manual > >>>>>>>> testing? > >>>>>>> > >>>>>>> Not presently, but we can work on improving the test. The current > >>>>>>> portal test was written as a regression test for a specific issue we > >>>>>>> had. It's quite hard to test completely end to end but we could do a > >>>>>>> lot better. > >>>>>>> > >>>>>>> I would quite like to spend some time in February or so working on our > >>>>>>> tests. > >>>>>> > >>>>>> Would it make sense to use openQA for this? Qubes OS uses openQA > >>>>>> and it works very well. openQA is written in Perl, but it’s the > >>>>>> best tool I know of for this. > >>>>> > >>>>> First blocker there would be packaging openQA in Nixpkgs. I do not > >>>>> personally relish the idea of doing that. > >>>> > >>>> Would it be possible to instead use a Fedora container? openQA is > >>>> packaged in Fedora. Qubes OS uses dedicated CI machines for openQA, > >>>> so I'm not worried about whether this would be permitted on your dev > >>>> box or the binary cache builders. > >>>> > >>>> I use Fedora for everything that isn't Spectrum-related dev work, > >>>> so I know how to maintain a Fedora system. That said, a container > >>>> shouldn't need much (if any) ongoing maintenance. > >>> > >>> I think the hermicity and bisectability of our build and tests are > >>> important properties worth preserving. We lose that if we start relying > >>> on an opaque container image. If an openQA update breaks something, > >>> it's not possible to easily figure out why. > >> Fedora container images contain an RPM database that can be used > >> to determine which packages changed. There will likely be many > >> packages that changed between images, but the same is true of Nixpkgs. > >> I totally agree that using a mutable Fedora system that is upgraded > >> in-place would be a mistake. > > > > This is not sufficient for bisectability, because I have no access to > > intermediate steps between the two images. > > How is Nixpkgs better in this regard? Is it because Nixpkgs only > changes one package at a time and has a linear history? > > Question for the Qubes developers: has Qubes OS ever ran into a > regression in openQA itself, and how hard was it to debug? It's a software, bugs happen. But they are rather rare (in the last 2 years I remember one, and it was rather easy to debug). More common case is a bug in the test suite itself. openQA has rather extensive git integration - it can record commit id of the test suite repo for each test, it can use specific repo/branch/commit of the test suite repo etc (so, you can test your test suite changes before merging them). We don't use this particular part that much, mostly because not too many people contribute to the openQA test suite (in most cases, we use openQA to run python tests and collect results - so, actual tests are stored elsewhere - together with the software being tested). There is also tracking of changes since last good test run, both on the openqa worker system and inside the system under test (SUT) - if your tests extract this info, see for example: https://openqa.qubes-os.org/tests/161428#investigation This last feature is very useful to see if a failure is related to some qubes change, or maybe some update in fedora/debian/etc. IIUC you sidestep this issue by using Nixpkgs, in which case those are not two separate things. Generally, I highly recommend openQA for full integration tests. But it can be also used for running other tests in openQA-prepared environment. BTW our openQA workers are running on openSUSE since that had best openQA packaging when we started using it (nowadays many other distributions have it packaged too). openSUSE isn't available in Qubes OS (yet?) and that never was an issue for using it for openQA. It's just part of the supporting infrastructure. - -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmk0S9EACgkQ24/THMrX 1yzSbAgAl516U28+mDrviL0bJP9llQgUHLa5YBW3WCBwgkmzImQu4EGLdf2c2cN/ Yk//vXPvngXCojrsZmk9M/0Nsksbf3Pewa5zLR4LEsnB5bFoaVP1ZFI75JWM+l0a RBTaC4Dbnj97x+LkHvxwzGDsFWtTdEE5V3oZ5iegWXsY7/MtI6LUIHxOA9YvszkT f/SYW4SEay8ipW1246vsuf7pddTdjEOz2SNchh/sxGJYLN68ZHJ+u0oYbgdTTv+h hHDMTIZBFsFQ6fZD+zTG6sFZHZtEHxW45cfmtSLHZNVqtk4aKKw2XTm6zUxZLQey ++GDjy19M0UOM3fbLNJf5OoOsDBoGw== =cZf/ -----END PGP SIGNATURE-----