Re: Test for portals passes even when file portal is broken

Alyssa Ross
2025-12-06T14:23:06Z

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 08:36, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>
>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>
>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>
>>>>>>> Is this a bug in the test? Is there a better alternative to manual
>>>>>>> testing?
>>>>>>
>>>>>> Not presently, but we can work on improving the test. The current
>>>>>> portal test was written as a regression test for a specific issue we
>>>>>> had. It's quite hard to test completely end to end but we could do a
>>>>>> lot better.
>>>>>>
>>>>>> I would quite like to spend some time in February or so working on our
>>>>>> tests.
>>>>>
>>>>> Would it make sense to use openQA for this? Qubes OS uses openQA
>>>>> and it works very well. openQA is written in Perl, but it’s the
>>>>> best tool I know of for this.
>>>>
>>>> First blocker there would be packaging openQA in Nixpkgs. I do not
>>>> personally relish the idea of doing that.
>>>
>>> Would it be possible to instead use a Fedora container? openQA is
>>> packaged in Fedora. Qubes OS uses dedicated CI machines for openQA,
>>> so I'm not worried about whether this would be permitted on your dev
>>> box or the binary cache builders.
>>>
>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>> so I know how to maintain a Fedora system. That said, a container
>>> shouldn't need much (if any) ongoing maintenance.
>>
>> I think the hermicity and bisectability of our build and tests are
>> important properties worth preserving. We lose that if we start relying
>> on an opaque container image. If an openQA update breaks something,
>> it's not possible to easily figure out why.
> Fedora container images contain an RPM database that can be used
> to determine which packages changed. There will likely be many
> packages that changed between images, but the same is true of Nixpkgs.
> I totally agree that using a mutable Fedora system that is upgraded
> in-place would be a mistake.

This is not sufficient for bisectability, because I have no access to
intermediate steps between the two images.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]