Demi Marie Obenour writes: > On 12/6/25 09:22, Alyssa Ross wrote: >> Demi Marie Obenour writes: >> >>> On 12/6/25 08:36, Alyssa Ross wrote: >>>> Demi Marie Obenour writes: >>>> >>>>> On 12/6/25 07:32, Alyssa Ross wrote: >>>>>> Demi Marie Obenour writes: >>>>>> >>>>>>> On 12/6/25 07:26, Alyssa Ross wrote: >>>>>>>> Demi Marie Obenour writes: >>>>>>>> >>>>>>>>> While trying to sandbox the file chooser portal, I broke it. >>>>>>>>> This caused files not to be saved, resulting in silent data loss. >>>>>>>>> Unfortunately, the integration test still passed. >>>>>>>>> >>>>>>>>> Is this a bug in the test? Is there a better alternative to manual >>>>>>>>> testing? >>>>>>>> >>>>>>>> Not presently, but we can work on improving the test. The current >>>>>>>> portal test was written as a regression test for a specific issue we >>>>>>>> had. It's quite hard to test completely end to end but we could do a >>>>>>>> lot better. >>>>>>>> >>>>>>>> I would quite like to spend some time in February or so working on our >>>>>>>> tests. >>>>>>> >>>>>>> Would it make sense to use openQA for this? Qubes OS uses openQA >>>>>>> and it works very well. openQA is written in Perl, but it’s the >>>>>>> best tool I know of for this. >>>>>> >>>>>> First blocker there would be packaging openQA in Nixpkgs. I do not >>>>>> personally relish the idea of doing that. >>>>> >>>>> Would it be possible to instead use a Fedora container? openQA is >>>>> packaged in Fedora. Qubes OS uses dedicated CI machines for openQA, >>>>> so I'm not worried about whether this would be permitted on your dev >>>>> box or the binary cache builders. >>>>> >>>>> I use Fedora for everything that isn't Spectrum-related dev work, >>>>> so I know how to maintain a Fedora system. That said, a container >>>>> shouldn't need much (if any) ongoing maintenance. >>>> >>>> I think the hermicity and bisectability of our build and tests are >>>> important properties worth preserving. We lose that if we start relying >>>> on an opaque container image. If an openQA update breaks something, >>>> it's not possible to easily figure out why. >>> Fedora container images contain an RPM database that can be used >>> to determine which packages changed. There will likely be many >>> packages that changed between images, but the same is true of Nixpkgs. >>> I totally agree that using a mutable Fedora system that is upgraded >>> in-place would be a mistake. >> >> This is not sufficient for bisectability, because I have no access to >> intermediate steps between the two images. > > How is Nixpkgs better in this regard? Is it because Nixpkgs only > changes one package at a time and has a linear history? Yes, exactly. It can be surprising to people used to traditional packaging systems how much of a productivity win this is. See also: https://gitlab.postmarketos.org/postmarketOS/postmarketos/-/issues/94