Test for portals passes even when file portal is broken

Test for portals passes even when file portal is broken

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 313 bytes --]

While trying to sandbox the file chooser portal, I broke it.
This caused files not to be saved, resulting in silent data loss.
Unfortunately, the integration test still passed.

Is this a bug in the test?  Is there a better alternative to manual
testing?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Test for portals passes even when file portal is broken

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> While trying to sandbox the file chooser portal, I broke it.
> This caused files not to be saved, resulting in silent data loss.
> Unfortunately, the integration test still passed.
>
> Is this a bug in the test?  Is there a better alternative to manual
> testing?

Not presently, but we can work on improving the test.  The current
portal test was written as a regression test for a specific issue we
had.  It's quite hard to test completely end to end but we could do a
lot better.

I would quite like to spend some time in February or so working on our
tests.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Test for portals passes even when file portal is broken

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 914 bytes --]

On 12/6/25 07:26, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> While trying to sandbox the file chooser portal, I broke it.
>> This caused files not to be saved, resulting in silent data loss.
>> Unfortunately, the integration test still passed.
>>
>> Is this a bug in the test?  Is there a better alternative to manual
>> testing?
> 
> Not presently, but we can work on improving the test.  The current
> portal test was written as a regression test for a specific issue we
> had.  It's quite hard to test completely end to end but we could do a
> lot better.
> 
> I would quite like to spend some time in February or so working on our
> tests.

Would it make sense to use openQA for this?  Qubes OS uses openQA
and it works very well.  openQA is written in Perl, but it’s the
best tool I know of for this.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Test for portals passes even when file portal is broken

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 07:26, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> While trying to sandbox the file chooser portal, I broke it.
>>> This caused files not to be saved, resulting in silent data loss.
>>> Unfortunately, the integration test still passed.
>>>
>>> Is this a bug in the test?  Is there a better alternative to manual
>>> testing?
>> 
>> Not presently, but we can work on improving the test.  The current
>> portal test was written as a regression test for a specific issue we
>> had.  It's quite hard to test completely end to end but we could do a
>> lot better.
>> 
>> I would quite like to spend some time in February or so working on our
>> tests.
>
> Would it make sense to use openQA for this?  Qubes OS uses openQA
> and it works very well.  openQA is written in Perl, but it’s the
> best tool I know of for this.

First blocker there would be packaging openQA in Nixpkgs.  I do not
personally relish the idea of doing that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Test for portals passes even when file portal is broken

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 1606 bytes --]

On 12/6/25 07:32, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> On 12/6/25 07:26, Alyssa Ross wrote:
>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>
>>>> While trying to sandbox the file chooser portal, I broke it.
>>>> This caused files not to be saved, resulting in silent data loss.
>>>> Unfortunately, the integration test still passed.
>>>>
>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>> testing?
>>>
>>> Not presently, but we can work on improving the test.  The current
>>> portal test was written as a regression test for a specific issue we
>>> had.  It's quite hard to test completely end to end but we could do a
>>> lot better.
>>>
>>> I would quite like to spend some time in February or so working on our
>>> tests.
>>
>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>> and it works very well.  openQA is written in Perl, but it’s the
>> best tool I know of for this.
> 
> First blocker there would be packaging openQA in Nixpkgs.  I do not
> personally relish the idea of doing that.

Would it be possible to instead use a Fedora container?  openQA is
packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
so I'm not worried about whether this would be permitted on your dev
box or the binary cache builders.

I use Fedora for everything that isn't Spectrum-related dev work,
so I know how to maintain a Fedora system.  That said, a container
shouldn't need much (if any) ongoing maintenance.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Test for portals passes even when file portal is broken

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2033 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 07:32, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>> Unfortunately, the integration test still passed.
>>>>>
>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>> testing?
>>>>
>>>> Not presently, but we can work on improving the test.  The current
>>>> portal test was written as a regression test for a specific issue we
>>>> had.  It's quite hard to test completely end to end but we could do a
>>>> lot better.
>>>>
>>>> I would quite like to spend some time in February or so working on our
>>>> tests.
>>>
>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>> and it works very well.  openQA is written in Perl, but it’s the
>>> best tool I know of for this.
>> 
>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>> personally relish the idea of doing that.
>
> Would it be possible to instead use a Fedora container?  openQA is
> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
> so I'm not worried about whether this would be permitted on your dev
> box or the binary cache builders.
>
> I use Fedora for everything that isn't Spectrum-related dev work,
> so I know how to maintain a Fedora system.  That said, a container
> shouldn't need much (if any) ongoing maintenance.

I think the hermicity and bisectability of our build and tests are
important properties worth preserving.  We lose that if we start relying
on an opaque container image.  If an openQA update breaks something,
it's not possible to easily figure out why.

I'm also not sure whether we'd be able to keep our tests as lightweight
and fast as they currently are with openQA.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Test for portals passes even when file portal is broken

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 3147 bytes --]

On 12/6/25 08:36, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> On 12/6/25 07:32, Alyssa Ross wrote:
>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>
>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>
>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>> Unfortunately, the integration test still passed.
>>>>>>
>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>> testing?
>>>>>
>>>>> Not presently, but we can work on improving the test.  The current
>>>>> portal test was written as a regression test for a specific issue we
>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>> lot better.
>>>>>
>>>>> I would quite like to spend some time in February or so working on our
>>>>> tests.
>>>>
>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>> best tool I know of for this.
>>>
>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>> personally relish the idea of doing that.
>>
>> Would it be possible to instead use a Fedora container?  openQA is
>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>> so I'm not worried about whether this would be permitted on your dev
>> box or the binary cache builders.
>>
>> I use Fedora for everything that isn't Spectrum-related dev work,
>> so I know how to maintain a Fedora system.  That said, a container
>> shouldn't need much (if any) ongoing maintenance.
> 
> I think the hermicity and bisectability of our build and tests are
> important properties worth preserving.  We lose that if we start relying
> on an opaque container image.  If an openQA update breaks something,
> it's not possible to easily figure out why.
Fedora container images contain an RPM database that can be used
to determine which packages changed.  There will likely be many
packages that changed between images, but the same is true of Nixpkgs.
I totally agree that using a mutable Fedora system that is upgraded
in-place would be a mistake.

> I'm also not sure whether we'd be able to keep our tests as lightweight
> and fast as they currently are with openQA.

openQA is an alternative to manual testing, not to unit or integration
testing.  openQA tests are very slow, but they are still faster,
cheaper, and much more repeatable than having a human do the work.
Qubes OS runs tests on many GitHub pull requests at once to amortize
the cost.

The workflow I envision is that openQA is run on patch series
immediately before committing them to the main branch.  There could
even be a separate branch from which openQA tests are run, and from
which patches are cherry-picked to main if openQA succeeds.

In any case, this a very large task in its own right, not something
that should be implemented for a single issue.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Test for portals passes even when file portal is broken

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 08:36, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>
>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>
>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>>> testing?
>>>>>>
>>>>>> Not presently, but we can work on improving the test.  The current
>>>>>> portal test was written as a regression test for a specific issue we
>>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>>> lot better.
>>>>>>
>>>>>> I would quite like to spend some time in February or so working on our
>>>>>> tests.
>>>>>
>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>>> best tool I know of for this.
>>>>
>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>>> personally relish the idea of doing that.
>>>
>>> Would it be possible to instead use a Fedora container?  openQA is
>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>>> so I'm not worried about whether this would be permitted on your dev
>>> box or the binary cache builders.
>>>
>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>> so I know how to maintain a Fedora system.  That said, a container
>>> shouldn't need much (if any) ongoing maintenance.
>> 
>> I think the hermicity and bisectability of our build and tests are
>> important properties worth preserving.  We lose that if we start relying
>> on an opaque container image.  If an openQA update breaks something,
>> it's not possible to easily figure out why.
> Fedora container images contain an RPM database that can be used
> to determine which packages changed.  There will likely be many
> packages that changed between images, but the same is true of Nixpkgs.
> I totally agree that using a mutable Fedora system that is upgraded
> in-place would be a mistake.

This is not sufficient for bisectability, because I have no access to
intermediate steps between the two images.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

openQA bisectability

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 2925 bytes --]

On 12/6/25 09:22, Alyssa Ross wrote:
> Demi Marie Obenour <demiobenour@gmail.com> writes:
> 
>> On 12/6/25 08:36, Alyssa Ross wrote:
>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>
>>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>
>>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>>
>>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>>
>>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>>>> testing?
>>>>>>>
>>>>>>> Not presently, but we can work on improving the test.  The current
>>>>>>> portal test was written as a regression test for a specific issue we
>>>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>>>> lot better.
>>>>>>>
>>>>>>> I would quite like to spend some time in February or so working on our
>>>>>>> tests.
>>>>>>
>>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>>>> best tool I know of for this.
>>>>>
>>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>>>> personally relish the idea of doing that.
>>>>
>>>> Would it be possible to instead use a Fedora container?  openQA is
>>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>>>> so I'm not worried about whether this would be permitted on your dev
>>>> box or the binary cache builders.
>>>>
>>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>>> so I know how to maintain a Fedora system.  That said, a container
>>>> shouldn't need much (if any) ongoing maintenance.
>>>
>>> I think the hermicity and bisectability of our build and tests are
>>> important properties worth preserving.  We lose that if we start relying
>>> on an opaque container image.  If an openQA update breaks something,
>>> it's not possible to easily figure out why.
>> Fedora container images contain an RPM database that can be used
>> to determine which packages changed.  There will likely be many
>> packages that changed between images, but the same is true of Nixpkgs.
>> I totally agree that using a mutable Fedora system that is upgraded
>> in-place would be a mistake.
> 
> This is not sufficient for bisectability, because I have no access to
> intermediate steps between the two images.

How is Nixpkgs better in this regard?  Is it because Nixpkgs only
changes one package at a time and has a linear history?

Question for the Qubes developers: has Qubes OS ever ran into a
regression in openQA itself, and how hard was it to debug?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [qubes-devel] openQA bisectability

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 3075 bytes --]

Demi Marie Obenour <demiobenour@gmail.com> writes:

> On 12/6/25 09:22, Alyssa Ross wrote:
>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>> 
>>> On 12/6/25 08:36, Alyssa Ross wrote:
>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>
>>>>> On 12/6/25 07:32, Alyssa Ross wrote:
>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>
>>>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
>>>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
>>>>>>>>
>>>>>>>>> While trying to sandbox the file chooser portal, I broke it.
>>>>>>>>> This caused files not to be saved, resulting in silent data loss.
>>>>>>>>> Unfortunately, the integration test still passed.
>>>>>>>>>
>>>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
>>>>>>>>> testing?
>>>>>>>>
>>>>>>>> Not presently, but we can work on improving the test.  The current
>>>>>>>> portal test was written as a regression test for a specific issue we
>>>>>>>> had.  It's quite hard to test completely end to end but we could do a
>>>>>>>> lot better.
>>>>>>>>
>>>>>>>> I would quite like to spend some time in February or so working on our
>>>>>>>> tests.
>>>>>>>
>>>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
>>>>>>> and it works very well.  openQA is written in Perl, but it’s the
>>>>>>> best tool I know of for this.
>>>>>>
>>>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
>>>>>> personally relish the idea of doing that.
>>>>>
>>>>> Would it be possible to instead use a Fedora container?  openQA is
>>>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
>>>>> so I'm not worried about whether this would be permitted on your dev
>>>>> box or the binary cache builders.
>>>>>
>>>>> I use Fedora for everything that isn't Spectrum-related dev work,
>>>>> so I know how to maintain a Fedora system.  That said, a container
>>>>> shouldn't need much (if any) ongoing maintenance.
>>>>
>>>> I think the hermicity and bisectability of our build and tests are
>>>> important properties worth preserving.  We lose that if we start relying
>>>> on an opaque container image.  If an openQA update breaks something,
>>>> it's not possible to easily figure out why.
>>> Fedora container images contain an RPM database that can be used
>>> to determine which packages changed.  There will likely be many
>>> packages that changed between images, but the same is true of Nixpkgs.
>>> I totally agree that using a mutable Fedora system that is upgraded
>>> in-place would be a mistake.
>> 
>> This is not sufficient for bisectability, because I have no access to
>> intermediate steps between the two images.
>
> How is Nixpkgs better in this regard?  Is it because Nixpkgs only
> changes one package at a time and has a linear history?

Yes, exactly.  It can be surprising to people used to traditional
packaging systems how much of a productivity win this is.  See also:

https://gitlab.postmarketos.org/postmarketOS/postmarketos/-/issues/94

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [qubes-devel] openQA bisectability

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Dec 06, 2025 at 09:47:51AM -0500, Demi Marie Obenour wrote:
> On 12/6/25 09:22, Alyssa Ross wrote:
> > Demi Marie Obenour <demiobenour@gmail.com> writes:
> > 
> >> On 12/6/25 08:36, Alyssa Ross wrote:
> >>> Demi Marie Obenour <demiobenour@gmail.com> writes:
> >>>
> >>>> On 12/6/25 07:32, Alyssa Ross wrote:
> >>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
> >>>>>
> >>>>>> On 12/6/25 07:26, Alyssa Ross wrote:
> >>>>>>> Demi Marie Obenour <demiobenour@gmail.com> writes:
> >>>>>>>
> >>>>>>>> While trying to sandbox the file chooser portal, I broke it.
> >>>>>>>> This caused files not to be saved, resulting in silent data loss.
> >>>>>>>> Unfortunately, the integration test still passed.
> >>>>>>>>
> >>>>>>>> Is this a bug in the test?  Is there a better alternative to manual
> >>>>>>>> testing?
> >>>>>>>
> >>>>>>> Not presently, but we can work on improving the test.  The current
> >>>>>>> portal test was written as a regression test for a specific issue we
> >>>>>>> had.  It's quite hard to test completely end to end but we could do a
> >>>>>>> lot better.
> >>>>>>>
> >>>>>>> I would quite like to spend some time in February or so working on our
> >>>>>>> tests.
> >>>>>>
> >>>>>> Would it make sense to use openQA for this?  Qubes OS uses openQA
> >>>>>> and it works very well.  openQA is written in Perl, but it’s the
> >>>>>> best tool I know of for this.
> >>>>>
> >>>>> First blocker there would be packaging openQA in Nixpkgs.  I do not
> >>>>> personally relish the idea of doing that.
> >>>>
> >>>> Would it be possible to instead use a Fedora container?  openQA is
> >>>> packaged in Fedora.  Qubes OS uses dedicated CI machines for openQA,
> >>>> so I'm not worried about whether this would be permitted on your dev
> >>>> box or the binary cache builders.
> >>>>
> >>>> I use Fedora for everything that isn't Spectrum-related dev work,
> >>>> so I know how to maintain a Fedora system.  That said, a container
> >>>> shouldn't need much (if any) ongoing maintenance.
> >>>
> >>> I think the hermicity and bisectability of our build and tests are
> >>> important properties worth preserving.  We lose that if we start relying
> >>> on an opaque container image.  If an openQA update breaks something,
> >>> it's not possible to easily figure out why.
> >> Fedora container images contain an RPM database that can be used
> >> to determine which packages changed.  There will likely be many
> >> packages that changed between images, but the same is true of Nixpkgs.
> >> I totally agree that using a mutable Fedora system that is upgraded
> >> in-place would be a mistake.
> > 
> > This is not sufficient for bisectability, because I have no access to
> > intermediate steps between the two images.
> 
> How is Nixpkgs better in this regard?  Is it because Nixpkgs only
> changes one package at a time and has a linear history?
> 
> Question for the Qubes developers: has Qubes OS ever ran into a
> regression in openQA itself, and how hard was it to debug?

It's a software, bugs happen. But they are rather rare (in the last 2
years I remember one, and it was rather easy to debug).

More common case is a bug in the test suite itself. openQA has rather
extensive git integration - it can record commit id of the test suite
repo for each test, it can use specific repo/branch/commit of the test
suite repo etc (so, you can test your test suite changes before merging
them). We don't use this particular part that much, mostly because not
too many people contribute to the openQA test suite (in most cases, we
use openQA to run python tests and collect results - so, actual tests
are stored elsewhere - together with the software being tested).

There is also tracking of changes since last good test run, both on the
openqa worker system and inside the system under test (SUT) - if your
tests extract this info, see for example:
https://openqa.qubes-os.org/tests/161428#investigation

This last feature is very useful to see if a failure is related to some
qubes change, or maybe some update in fedora/debian/etc. IIUC you
sidestep this issue by using Nixpkgs, in which case those are not two
separate things.

Generally, I highly recommend openQA for full integration tests. But it
can be also used for running other tests in openQA-prepared environment.

BTW our openQA workers are running on openSUSE since that had best
openQA packaging when we started using it (nowadays many other
distributions have it packaged too). openSUSE isn't available in Qubes
OS (yet?) and that never was an issue for using it for openQA. It's just
part of the supporting infrastructure.

- -- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmk0S9EACgkQ24/THMrX
1yzSbAgAl516U28+mDrviL0bJP9llQgUHLa5YBW3WCBwgkmzImQu4EGLdf2c2cN/
Yk//vXPvngXCojrsZmk9M/0Nsksbf3Pewa5zLR4LEsnB5bFoaVP1ZFI75JWM+l0a
RBTaC4Dbnj97x+LkHvxwzGDsFWtTdEE5V3oZ5iegWXsY7/MtI6LUIHxOA9YvszkT
f/SYW4SEay8ipW1246vsuf7pddTdjEOz2SNchh/sxGJYLN68ZHJ+u0oYbgdTTv+h
hHDMTIZBFsFQ6fZD+zTG6sFZHZtEHxW45cfmtSLHZNVqtk4aKKw2XTm6zUxZLQey
++GDjy19M0UOM3fbLNJf5OoOsDBoGw==
=cZf/
-----END PGP SIGNATURE-----