openQA bisectability

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 2925 bytes --]

On 12/6/25 09:22, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> On 12/6/25 08:36, Alyssa Ross wrote:
>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>
>>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>
>>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>>
>>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>>
>>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>>>> testing?
>>>>>>>
>>>>>>> Not presently, but we can work on improving the test.  The current
>>>>>>> portal test was written as a regression test for a specific issue we
>>>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>>>> lot better.
>>>>>>>
>>>>>>> I would quite like to spend some time in February or so working on our
>>>>>>> tests.
>>>>>>
>>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>>>> best tool I know of for this.
>>>>>
>>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>>>> personally relish the idea of doing that.
>>>>
>>>> Would it be possible to instead use a Fedora container?  openQA is
>>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>>>> so I'm not worried about whether this would be permitted on your dev
>>>> box or the binary cache builders.
>>>>
>>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>>> so I know how to maintain a Fedora system.  That said, a container
>>>> shouldn't need much (if any) ongoing maintenance.
>>>
>>> I think the hermicity and bisectability of our build and tests are
>>> important properties worth preserving.  We lose that if we start relying
>>> on an opaque container image.  If an openQA update breaks something,
>>> it's not possible to easily figure out why.
>> Fedora container images contain an RPM database that can be used
>> to determine which packages changed.  There will likely be many
>> packages that changed between images, but the same is true of Nixpkgs.
>> I totally agree that using a mutable Fedora system that is upgraded
>> in-place would be a mistake.
> 
> This is not sufficient for bisectability, because I have no access to
> intermediate steps between the two images.

How is Nixpkgs better in this regard?  Is it because Nixpkgs only
changes one package at a time and has a linear history?

Question for the Qubes developers: has Qubes OS ever ran into a
regression in openQA itself, and how hard was it to debug?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]