Re: Test for portals passes even when file portal is broken

[Alyssa Ross <hi@alyssa.is>]

[-- Attachment #1: Type: text/plain, Size: 2033 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 07:32, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>> Unfortunately, the integration test still passed.
>>>>>
>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>> testing?
>>>>
>>>> Not presently, but we can work on improving the test.  The current
>>>> portal test was written as a regression test for a specific issue we
>>>> had.  It's quite hard to test completely end to end but we could do a
>>>> lot better.
>>>>
>>>> I would quite like to spend some time in February or so working on our
>>>> tests.
>>>
>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>> and it works very well.  openQA is written in Perl, but it’s the
>>> best tool I know of for this.
>> 
>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>> personally relish the idea of doing that.
>
> Would it be possible to instead use a Fedora container?  openQA is
> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
> so I'm not worried about whether this would be permitted on your dev
> box or the binary cache builders.
>
> I use Fedora for everything that isn't Spectrum-related dev work,
> so I know how to maintain a Fedora system.  That said, a container
> shouldn't need much (if any) ongoing maintenance.

I think the hermicity and bisectability of our build and tests are
important properties worth preserving.  We lose that if we start relying
on an opaque container image.  If an openQA update breaks something,
it's not possible to easily figure out why.

I'm also not sure whether we'd be able to keep our tests as lightweight
and fast as they currently are with openQA.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]