Re: [qubes-devel] openQA bisectability

[Alyssa Ross <hi@alyssa.is>]

[-- Attachment #1: Type: text/plain, Size: 3075 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 09:22, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> On 12/6/25 08:36, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>
>>>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>>>
>>>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>>>
>>>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>>>>> testing?
>>>>>>>>
>>>>>>>> Not presently, but we can work on improving the test.  The current
>>>>>>>> portal test was written as a regression test for a specific issue we
>>>>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>>>>> lot better.
>>>>>>>>
>>>>>>>> I would quite like to spend some time in February or so working on our
>>>>>>>> tests.
>>>>>>>
>>>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>>>>> best tool I know of for this.
>>>>>>
>>>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>>>>> personally relish the idea of doing that.
>>>>>
>>>>> Would it be possible to instead use a Fedora container?  openQA is
>>>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>>>>> so I'm not worried about whether this would be permitted on your dev
>>>>> box or the binary cache builders.
>>>>>
>>>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>>>> so I know how to maintain a Fedora system.  That said, a container
>>>>> shouldn't need much (if any) ongoing maintenance.
>>>>
>>>> I think the hermicity and bisectability of our build and tests are
>>>> important properties worth preserving.  We lose that if we start relying
>>>> on an opaque container image.  If an openQA update breaks something,
>>>> it's not possible to easily figure out why.
>>> Fedora container images contain an RPM database that can be used
>>> to determine which packages changed.  There will likely be many
>>> packages that changed between images, but the same is true of Nixpkgs.
>>> I totally agree that using a mutable Fedora system that is upgraded
>>> in-place would be a mistake.
>> 
>> This is not sufficient for bisectability, because I have no access to
>> intermediate steps between the two images.
>
> How is Nixpkgs better in this regard?  Is it because Nixpkgs only
> changes one package at a time and has a linear history?

Yes, exactly.  It can be surprising to people used to traditional
packaging systems how much of a productivity win this is.  See also:

https://gitlab.postmarketos.org/postmarketOS/postmarketos/-/issues/94

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]