Comparison to Qubes OS

Comparison to Qubes OS

[joweill]

Hi there,

I just discovered this project and I'm really excited about it! I've long waited for an OS that combines Qubes-like compartmentalization with the reproducibility of Nix/GUIX.

I'm trying to understand what this project aims to improve over Qubes, other than the integration of Nix (which I do think is really important!). I read the motivation page [1], but I'm not yet convinced by most
of the points mentioned that relate to Qubes. Going over each usability issue mentioned in the motivation doc:

- "Hardware compatibility is extremely limited": I don't believe this is really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy support these. Is there anything I'm missing?
- "People are reluctant to use Xen on their computer for power management etc. reasons." Can you elaborate on these issues?
- I know that Qubes considered using KVM and decided against it for security reasons [2]. My understanding is that the downside of this decision is the limited hardware support, which is one of the things that Spectrum views as an opportunity for improvement. Can you elaborate on this decision?
- "VMs are heavy": How will Spectrum improve on this without sacrificing security?
- "GUI applications are buggy, command line tools are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be
  better in the case of Spectrum which is a new project with one developer.

More generally, I'm wondering whether this projects' goals couldn't be better achieved by trying to work with the Qubes developers to integrate Nix. It may very well be that they would reject it for
some reason, but then the logical next step would be to fork Qubes.
Have you reached out to the Qubes developers?

Thanks in advance!

## References

[1] https://spectrum-os.org/motivation.html
[2] https://www.qubes-os.org/faq/#why-does-qubes-use-xen-instead-of-kvm-or-some-other-hypervisor

Re: Comparison to Qubes OS

[Michał "rysiek" Woźniak]

[-- Attachment #1.1: Type: text/plain, Size: 3642 bytes --]

Hi!

A QubesOS user here for about a year and a half, let's see if I can help out here.

On 6/12/20 11:06 AM, joweill@icloud.com wrote:
> Going over each usability issue mentioned in the motivation doc:
> 
> - "Hardware compatibility is extremely limited": I don't believe this is really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy support these. Is there anything I'm missing?

I got a shiny new Thinkpad T490 a few months ago. 3d acceleration (KDE user
here, I demand my wobbly windows!) is simply not available, because the dom0
system is too old.

My other laptop, a T470, was specifically selected for QubesOS, and there were
still issues (for instance, disabling Thuderbolt got me almost double the
battery life).

Generally speaking, one can buy almost any laptop today and expect it to mostly
work with plain GNU/Linux. However, most might not even be able to boot QubesOS.

> - "People are reluctant to use Xen on their computer for power management etc. reasons." Can you elaborate on these issues?

The T470 had easily a 10-12h battery life on plain Kubuntu. On Qubes, 4-5h is
maximum I can squeeze out of it, and that's *after* the Tunderbolt fix.

Running virtual machines is extremely resource-intensive, there's no way around it.

> - I know that Qubes considered using KVM and decided against it for security reasons [2]. My understanding is that the downside of this decision is the limited hardware support, which is one of the things that Spectrum views as an opportunity for improvement. Can you elaborate on this decision?

Can't speak for the developers, but the way I see Spectrum is as a compromise
between regular GNU/Linux distro (with all the related security problems) and
QubesOS (with the limited hardware support and

> - "VMs are heavy": How will Spectrum improve on this without sacrificing security?

I'll leave this to the developers, but will say that I expect *some* security to
be sacrificed. There are always trade-offs.

I feel one needs to be an expert to use QubesOS, but a regular user (with some
basic training) can use a Mint or Ubuntu-based system. And I think it makes a
lot of sense to offer a middle ground.

> - "GUI applications are buggy, command line tools are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be
>   better in the case of Spectrum which is a new project with one developer.

That's a fair point. Things to consider:
1. *probably* certain things can be easier (thus, less bug-prone) in SpectrumOS
than in QubesOS (kvm easier to work with than Xen, bigger potential community of
users and developers due to improved hardware support, etc);
2. perhaps some QubesOS tools could be used in SpectrumOS, thus limiting the
amount of work needed

> More generally, I'm wondering whether this projects' goals couldn't be better achieved by trying to work with the Qubes developers to integrate Nix. It may very well be that they would reject it for
> some reason, but then the logical next step would be to fork Qubes.

My feel is that QubesOS and SpectrumOS might have a bit different threat models
in mind, and thus things that make sense for SpectrumOS (like using kvm) are a
no-go for Qubes. But that's just guesswork on my part.

> Have you reached out to the Qubes developers?
> 
> Thanks in advance!
> 
> ## References
> 
> [1] https://spectrum-os.org/motivation.html
> [2] https://www.qubes-os.org/faq/#why-does-qubes-use-xen-instead-of-kvm-or-some-other-hypervisor
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Comparison to Qubes OS

[infokiller ​]

Thanks for a useful reply Michał!
Some responses below.

Michał "rysiek" Woźniak wrote:
> Hi!
> 
> A QubesOS user here for about a year and a half, let's see if I can help out here.
> 
> On 6/12/20 11:06 AM, joweill(a)icloud.com wrote:
> >   Going over each usability issue mentioned in the
> > motivation doc:
> >  
> >  - "Hardware compatibility is extremely limited": I don't believe this is
> > really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy
> > support these. Is there anything I'm missing? 
> I got a shiny new Thinkpad T490 a few months ago. 3d acceleration (KDE user
> here, I demand my wobbly windows!) is simply not available, because the dom0
> system is too old.
> 
> My other laptop, a T470, was specifically selected for QubesOS, and there were
> still issues (for instance, disabling Thuderbolt got me almost double the
> battery life).
> 
> Generally speaking, one can buy almost any laptop today and expect it to mostly
> work with plain GNU/Linux. However, most might not even be able to boot QubesOS.
> 
> >   - "People are reluctant to use Xen on their
> > computer for power management etc. reasons." Can you elaborate on these issues?
> > 
> The T470 had easily a 10-12h battery life on plain Kubuntu. On Qubes, 4-5h is
> maximum I can squeeze out of it, and that's *after* the Tunderbolt fix.
> 
> Running virtual machines is extremely resource-intensive, there's no way around it.
But if the issues stem from running VMs (and not switching from Xen), they won't be resolved with Spectrum's current design, right?
> 
> >   - I know that Qubes considered using KVM and decided
> > against it for security reasons [2]. My understanding is that the downside of this
> > decision is the limited hardware support, which is one of the things that Spectrum views
> > as an opportunity for improvement. Can you elaborate on this decision? 
> Can't speak for the developers, but the way I see Spectrum is as a compromise
> between regular GNU/Linux distro (with all the related security problems) and
> QubesOS (with the limited hardware support and
> 
> >   - "VMs are heavy": How will Spectrum improve
> > on this without sacrificing security? 
> I'll leave this to the developers, but will say that I expect *some* security to
> be sacrificed. There are always trade-offs.
> 
> I feel one needs to be an expert to use QubesOS, but a regular user (with some
> basic training) can use a Mint or Ubuntu-based system. And I think it makes a
> lot of sense to offer a middle ground.
Agreed, but I think the current design of Spectrum may improve Qubes' hardware issues, but not the other issues the doc mentions. Possibly switching to containers (which something like gVisor) may solve some of the other issues of Qubes, though that would further degrade security.
> 
> >   - "GUI applications are buggy, command line tools
> > are mostly undocumented": I assume that the reason for this is the lack of resources
> > the Qubes project has. However, I don't see how this will be be
> >    better in the case of Spectrum which is a new project with one developer. 
> That's a fair point. Things to consider:
> 1. *probably* certain things can be easier (thus, less bug-prone) in SpectrumOS
> than in QubesOS (kvm easier to work with than Xen, bigger potential community of
> users and developers due to improved hardware support, etc);
> 2. perhaps some QubesOS tools could be used in SpectrumOS, thus limiting the
> amount of work needed
> 
> >   More generally, I'm wondering whether this
> > projects' goals couldn't be better achieved by trying to work with the Qubes
> > developers to integrate Nix. It may very well be that they would reject it for
> >  some reason, but then the logical next step would be to fork Qubes. 
> My feel is that QubesOS and SpectrumOS might have a bit different threat models
> in mind, and thus things that make sense for SpectrumOS (like using kvm) are a
> no-go for Qubes. But that's just guesswork on my part.
> 
> >   Have you reached out to the Qubes developers?
> >  
> >  Thanks in advance!
> >  
> >  ## References
> >  
> >  [1] https://spectrum-os.org/motivation.html
> >  [2]
> > https://www.qubes-os.org/faq/#why-does-qubes-use-xen-instead-of-kvm-or-some…
> >

Re: Comparison to Qubes OS

[Michał "rysiek" Woźniak]

[-- Attachment #1.1: Type: text/plain, Size: 1217 bytes --]

Hey,

On 6/12/20 11:54 AM, infokiller wrote:
> Michał "rysiek" Woźniak wrote:
>> Running virtual machines is extremely resource-intensive, there's no way around it.
>
> But if the issues stem from running VMs (and not switching from Xen), they won't be resolved with Spectrum's current design, right?

Right you are about the performance issues *I think*. I would however expect
hardware compatibility to improve with kvm.

But I'd love somebody more knowledgeable to shed some light here.

>> I feel one needs to be an expert to use QubesOS, but a regular user (with some
>> basic training) can use a Mint or Ubuntu-based system. And I think it makes a
>> lot of sense to offer a middle ground.
>
> Agreed, but I think the current design of Spectrum may improve Qubes' hardware issues, but not the other issues the doc mentions. Possibly switching to containers (which something like gVisor) may solve some of the other issues of Qubes, though that would further degrade security.

Fun fact, containers were in fact initially considered. :-)

Here's a good blogpost from Alyssa on why move to kvm, and the advantages of it
over Xen: https://alyssa.is/back-from-cccamp-2019/

--
Best
r


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Comparison to Qubes OS

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 4327 bytes --]

Michał "rysiek" Woźniak <rysiek@hackerspace.pl> writes:

> On 6/12/20 11:54 AM, infokiller wrote:
>> Michał "rysiek" Woźniak wrote:
>>> Running virtual machines is extremely resource-intensive, there's no way around it.
>>
>> But if the issues stem from running VMs (and not switching from Xen), they won't be resolved with Spectrum's current design, right?
>
> Right you are about the performance issues *I think*. I would however expect
> hardware compatibility to improve with kvm.
>
> But I'd love somebody more knowledgeable to shed some light here.

I am by no means a Xen expert (and so the following could be totally
wrong, and I would appreciate being corrected), but my understanding is
that Linux is likely to provide better power management than Xen due to
being better optimised, and not being as focused as Xen on large data
center deployments.

Additionally, there are interesting Linux virtualization features that
may allow for reduced overall hardware requirements by securely sharing
resources between VMs.  For example, it may be possible to share
identical read-only memory pages between VMs.  There's not necessarily
any reason that running identical copies of Firefox in two different VMs
should require two copies of Firefox in host memory.  The most promising
application of this principle at the moment in virtio-fs[1][2], which
supports this page sharing.  DAX[3] may also be of interest.  Of course,
there will be security considerations when deciding whether to adopt
these systems, but the point is that this sort of development just isn't
happening with Xen.  It's all focused on the KVM ecosystem.

[1]: https://virtio-fs.gitlab.io/
[2]: https://lwn.net/Articles/788333/
[3]: https://lwn.net/Articles/610174/

Hardware compatibility should certainly be better with KVM.  Linux
has extremely diverse hardware support, and a much larger user base than
Xen, especially on consumer hardware.  I'd be confident in saying most
relatively recent consumer PC hardware will support Linux reasonably
well (a notable exception of course being Nvidea GPUs), and I'd be
surprised if Xen worked well on anywhere near as many devices.

>>> I feel one needs to be an expert to use QubesOS, but a regular user (with some
>>> basic training) can use a Mint or Ubuntu-based system. And I think it makes a
>>> lot of sense to offer a middle ground.
>>
>> Agreed, but I think the current design of Spectrum may improve Qubes' hardware issues, but not the other issues the doc mentions. Possibly switching to containers (which something like gVisor) may solve some of the other issues of Qubes, though that would further degrade security.
>
> Fun fact, containers were in fact initially considered. :-)
>
> Here's a good blogpost from Alyssa on why move to kvm, and the advantages of it
> over Xen: https://alyssa.is/back-from-cccamp-2019/

Indeed, Spectrum was originally conceived as a Qubes-style operating
system that used containers instead of VMs.  The reason the focus
shifted from containers to KVM is that as I researched the topic, I
became less confident in the ability of containers to offer meaningful
security, and learned that the overhead of VMs doesn't have to be as big
as I thought.  Of particular inspiration was "My VM is Ligther (and
Faster) than your Container"[4], which demonstrates Xen-based virtual
machines with resource requirements roughly equivalent to containers.

I also think that Nix might put Spectrum in a unique position to provide
lightweight virtual machines.  Conventionally, VMs run full-featured
Linux distributions, with all the memory and disk requirements that
entails.  It's not usually feasible to build tailored VM images running
only the software that is critical to the task of the VM.  With Nix,
however, it suddenly is.  A big part of the appeal of containers, I
think, is that the tooling makes it easy to create small images focused
on a single task.  VM tooling has never really provided this.  It's a
sort of vicious cycle of assuming VMs are heavy and therefore it doesn't
make sense to have a VM dedicated to a single task, which means that
tooling for creating such a VM isn't developed, which means that VMs
stay heavy.

[4]: http://cnp.neclab.eu/projects/lightvm/lightvm.pdf

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Comparison to Qubes OS

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 3642 bytes --]

> Hi there,
>
> I just discovered this project and I'm really excited about it! I've
> long waited for an OS that combines Qubes-like compartmentalization
> with the reproducibility of Nix/GUIX.

Hi!  Thank you for getting in touch. :)

> I'm trying to understand what this project aims to improve over Qubes, other than the integration of Nix (which I do think is really important!). I read the motivation page [1], but I'm not yet convinced by most
> of the points mentioned that relate to Qubes. Going over each usability issue mentioned in the motivation doc:
>
> - "Hardware compatibility is extremely limited": I don't believe this is really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy support these. Is there anything I'm missing?
> - "People are reluctant to use Xen on their computer for power management etc. reasons." Can you elaborate on these issues?
> - I know that Qubes considered using KVM and decided against it for security reasons [2]. My understanding is that the downside of this decision is the limited hardware support, which is one of the things that Spectrum views as an opportunity for improvement. Can you elaborate on this decision?
> - "VMs are heavy": How will Spectrum improve on this without sacrificing security?

I already talked about Xen vs KVM elsewhere in the thread, but wanted to
say that the impression I've got from talking to several Qubes
developers is that they are at this point far less enthusiastic about
Xen than their documentation might suggest.

Additionally, a couple of things that the Qubes Architecture
Specification[1] mentions as being benefits of Xen over KVM are now, I
believe, also available in KVM.  Specifically, I believe it is possible
to use PV drivers with KVM (although I wonder if maybe there is a
terminology mismatch here?), and I believe it will be at least partially
possible to have something resembling driver domains, although this is
an area of active R&D within Spectrum.  Other mentioned advantages of
Xen, such as auditability, remain, of course.

[1]: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf

> - "GUI applications are buggy, command line tools are mostly
>   undocumented": I assume that the reason for this is the lack of
>   resources the Qubes project has. However, I don't see how this will be
>   be better in the case of Spectrum which is a new project with one
>   developer.

My understanding is that a lot of the instability I've encountered with
Qubes's tools comes down to some severe technical debt with their
inter-VM communication system.  This is likely something that is very
difficult to fix, but easy to learn from.  Being a new project allows
Spectrum to learn from Qubes' mistakes.

> More generally, I'm wondering whether this projects' goals couldn't be
> better achieved by trying to work with the Qubes developers to
> integrate Nix. It may very well be that they would reject it for some
> reason, but then the logical next step would be to fork Qubes.  Have
> you reached out to the Qubes developers?

I've had the pleasure of speaking with several Qubes developers on
multiple occasions.  I also know that there is work being done to
support NixOS templates in Qubes.  As I see it, though, the real benefit
of Nix here is that it would allow defining the whole system in one
single Nix configuration.  Doing this in Qubes would require big
fundamental changes to how Qubes works.  I believe the idea has been
brought up to the Qubes developers, and as I recall they were not keen.
I believe there is room in this space for Qubes and Spectrum to coexist.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Comparison to Qubes OS

[infokiller ​]

Thanks for your detailed and well put responses!

Alyssa Ross wrote:
> >   Hi there,
> > 
> >  I just discovered this project and I'm really excited about it! I've
> >  long waited for an OS that combines Qubes-like compartmentalization
> >  with the reproducibility of Nix/GUIX. 
> Hi!  Thank you for getting in touch. :)
> 
> >   I'm trying to understand what this project aims to
> > improve over Qubes, other than the integration of Nix (which I do think is really
> > important!). I read the motivation page [1], but I'm not yet convinced by most
> >  of the points mentioned that relate to Qubes. Going over each usability issue mentioned
> > in the motivation doc:
> > 
> >  - "Hardware compatibility is extremely limited": I don't believe this is
> > really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy
> > support these. Is there anything I'm missing?
> >  - "People are reluctant to use Xen on their computer for power management etc.
> > reasons." Can you elaborate on these issues?
> >  - I know that Qubes considered using KVM and decided against it for security reasons [2].
> > My understanding is that the downside of this decision is the limited hardware support,
> > which is one of the things that Spectrum views as an opportunity for improvement. Can you
> > elaborate on this decision?
> >  - "VMs are heavy": How will Spectrum improve on this without sacrificing
> > security? 
> I already talked about Xen vs KVM elsewhere in the thread, but wanted to
> say that the impression I've got from talking to several Qubes
> developers is that they are at this point far less enthusiastic about
> Xen than their documentation might suggest.
> 
> Additionally, a couple of things that the Qubes Architecture
> Specification[1] mentions as being benefits of Xen over KVM are now, I
> believe, also available in KVM.  Specifically, I believe it is possible
> to use PV drivers with KVM (although I wonder if maybe there is a
> terminology mismatch here?), and I believe it will be at least partially
> possible to have something resembling driver domains, although this is
> an area of active R&D within Spectrum.  Other mentioned advantages of
> Xen, such as auditability, remain, of course.
> 
> [1]: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf
> 
> >   - "GUI applications are buggy, command line tools
> > are mostly
> >    undocumented": I assume that the reason for this is the lack of
> >    resources the Qubes project has. However, I don't see how this will be
> >    be better in the case of Spectrum which is a new project with one
> >    developer. 
> My understanding is that a lot of the instability I've encountered with
> Qubes's tools comes down to some severe technical debt with their
> inter-VM communication system.  This is likely something that is very
> difficult to fix, but easy to learn from.  Being a new project allows
> Spectrum to learn from Qubes' mistakes.
I agree, but I'm still not convinced this will be different in the case of Spectrum, having even more limited dev resources than Qubes'. I mean, if the Qubes folks could fix these issues without a
huge effort, even if it
meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation.

> 
> >   More generally, I'm wondering whether this
> > projects' goals couldn't be
> >  better achieved by trying to work with the Qubes developers to
> >  integrate Nix. It may very well be that they would reject it for some
> >  reason, but then the logical next step would be to fork Qubes.  Have
> >  you reached out to the Qubes developers? 
> I've had the pleasure of speaking with several Qubes developers on
> multiple occasions.  I also know that there is work being done to
> support NixOS templates in Qubes.  
Do you have any link you can share to work on Nix template being used in Qubes?

> As I see it, though, the real benefit
> of Nix here is that it would allow defining the whole system in one
> single Nix configuration.  Doing this in Qubes would require big
> fundamental changes to how Qubes works.  I believe the idea has been
> brought up to the Qubes developers, and as I recall they were not keen.
> I believe there is room in this space for Qubes and Spectrum to coexist.

There's definitely a place for both efforts, but I'm just wondering if it would not be better to implement Spectrum's ideas while working more closely with Qubes. For example, creating a git branch or
a fork that replaces Xen with KVM and integrates Nix. Now, that may very be not the best way to make progress on this project, because ramping up on the Qubes codebase will be an inefficient use of your time, or because you want to first experiment with some ideas in isolation, or for any other reason. However, the potential benefits of such a collaboration are, in my opinion, too big to pass on without strongly pursuing it. Just to name a few:
- You will get feedback from top tier security experts
- Qubes already has strong credibility which can be a huge driver of adoption, especially if your project will be endorsed by Qubes devs
- You will be in a better position to integrate some of the tools of Qubes, which you likely would need to do, and it is less likely that changes in the Qubes tooling will break your project (if you will indeed integrate them)
- You are less likely to be asked time and again "Why not Qubes"? :)

So, why not start a public discussion in Qubes' mailing list on issue tracker to figure out what is needed to accomplish Spectrum's goals?
It will probably turn out that you made the right decision by starting a separate project, but at the very least: 
- You'll may get attention from people who can contribute to Spectrum
- The issues involved with be publicly documented and searchable for future generations

Re: Comparison to Qubes OS

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 5116 bytes --]

>> >   - "GUI applications are buggy, command line tools
>> > are mostly
>> >    undocumented": I assume that the reason for this is the lack of
>> >    resources the Qubes project has. However, I don't see how this will be
>> >    be better in the case of Spectrum which is a new project with one
>> >    developer. 
>> My understanding is that a lot of the instability I've encountered with
>> Qubes's tools comes down to some severe technical debt with their
>> inter-VM communication system.  This is likely something that is very
>> difficult to fix, but easy to learn from.  Being a new project allows
>> Spectrum to learn from Qubes' mistakes.
> I agree, but I'm still not convinced this will be different in the
> case of Spectrum, having even more limited dev resources than
> Qubes'. I mean, if the Qubes folks could fix these issues without a
> huge effort, even if it meant rewriting all the inter-VM communication
> tools, they probably would. If they didn't, I assume this is because
> this is just a huge undertaking (as is the whole project), and they're
> busy with other work which has higher priority. I would assume that
> you will end up in exactly the same situation.

Maybe eventually, but the flexibility that comes with being a new
project still gives us a significant opportunity to advance the state of
the art before that happens.  At that point, maybe something else will
come along to innovate free of all the technical debt Spectrum has
acquired by then, and then the state of the art will advance even
further.  Such is the nature of software development.

>> >   More generally, I'm wondering whether this
>> > projects' goals couldn't be
>> >  better achieved by trying to work with the Qubes developers to
>> >  integrate Nix. It may very well be that they would reject it for some
>> >  reason, but then the logical next step would be to fork Qubes.  Have
>> >  you reached out to the Qubes developers? 
>> I've had the pleasure of speaking with several Qubes developers on
>> multiple occasions.  I also know that there is work being done to
>> support NixOS templates in Qubes.  
> Do you have any link you can share to work on Nix template being used in Qubes?

This was the last I heard of it:

https://hackmd.shackspace.de/qubes-nixos

>> As I see it, though, the real benefit
>> of Nix here is that it would allow defining the whole system in one
>> single Nix configuration.  Doing this in Qubes would require big
>> fundamental changes to how Qubes works.  I believe the idea has been
>> brought up to the Qubes developers, and as I recall they were not keen.
>> I believe there is room in this space for Qubes and Spectrum to coexist.
>
> There's definitely a place for both efforts, but I'm just wondering if
> it would not be better to implement Spectrum's ideas while working
> more closely with Qubes. For example, creating a git branch or a fork
> that replaces Xen with KVM and integrates Nix. Now, that may very be
> not the best way to make progress on this project, because ramping up
> on the Qubes codebase will be an inefficient use of your time, or
> because you want to first experiment with some ideas in isolation, or
> for any other reason. However, the potential benefits of such a
> collaboration are, in my opinion, too big to pass on without strongly
> pursuing it. Just to name a few:
>
> - You will get feedback from top tier security experts
> - Qubes already has strong credibility which can be a huge driver of
>   adoption, especially if your project will be endorsed by Qubes devs
> - You will be in a better position to integrate some of the tools of
>   Qubes, which you likely would need to do, and it is less likely
>   that changes in the Qubes tooling will break your project (if you
>   will indeed integrate them)
> - You are less likely to be asked time and again "Why not Qubes"? :)
>
> So, why not start a public discussion in Qubes' mailing list on issue
> tracker to figure out what is needed to accomplish Spectrum's goals?
>
> It will probably turn out that you made the right decision by starting
> a separate project, but at the very least:
> - You'll may get attention from people who can contribute to Spectrum
> - The issues involved with be publicly documented and searchable for
>   future generations

I think you maybe don't appreciate just how huge an undertaking this
would be.  There is so much that would have to change about how Qubes
works that I think you'd end up having to reimplement most of it
anyway, but you'd be doing it bit by bit, never having the opportunity
to consider the system as a whole.

At the end of the day I just don't believe that trying to shoehorn these
changes into Qubes is the best way to make progress.  It might well be
valuable to try that, but even so it would make much more sense for
somebody who believes in that approach to dedicate the huge amount of
effort required to attempt it, rather than me.  This could be another
effort that could be pursued in parallel to my work on Spectrum.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Comparison to Qubes OS

[Michał "rysiek" Woźniak]

[-- Attachment #1.1: Type: text/plain, Size: 1562 bytes --]

On 6/14/20 9:27 PM, Alyssa Ross wrote:
>> So, why not start a public discussion in Qubes' mailing list on issue
>> tracker to figure out what is needed to accomplish Spectrum's goals?
>>
>> It will probably turn out that you made the right decision by starting
>> a separate project, but at the very least:
>> - You'll may get attention from people who can contribute to Spectrum
>> - The issues involved with be publicly documented and searchable for
>>   future generations
> 
> I think you maybe don't appreciate just how huge an undertaking this
> would be.  There is so much that would have to change about how Qubes
> works that I think you'd end up having to reimplement most of it
> anyway, but you'd be doing it bit by bit, never having the opportunity
> to consider the system as a whole.

Plus, SpectrumOS does not have to deal with backwards compatibility. If QubesOS
developers were to start implementing these changes, they would constantly have
to deal with trade-offs between ease of implementing them and the cost of
breaking backwards compatibility.

> At the end of the day I just don't believe that trying to shoehorn these
> changes into Qubes is the best way to make progress.  It might well be
> valuable to try that, but even so it would make much more sense for
> somebody who believes in that approach to dedicate the huge amount of
> effort required to attempt it, rather than me.  This could be another
> effort that could be pursued in parallel to my work on Spectrum.

Very much this.

--
Best,
r


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Comparison to Qubes OS

[infokiller ​]

Michał "rysiek" Woźniak wrote:
> On 6/14/20 9:27 PM, Alyssa Ross wrote:
> >    So, why not
> > start a public discussion in Qubes' mailing list on issue
> >  tracker to figure out what is needed to accomplish Spectrum's goals?
> > 
> >  It will probably turn out that you made the right decision by starting
> >  a separate project, but at the very least:
> >  - You'll may get attention from people who can contribute to Spectrum
> >  - The issues involved with be publicly documented and searchable for
> >    future generations  
> >  I think you maybe don't appreciate just how huge an undertaking this
> >  would be.  There is so much that would have to change about how Qubes
> >  works that I think you'd end up having to reimplement most of it
> >  anyway, but you'd be doing it bit by bit, never having the opportunity
> >  to consider the system as a whole. 
> Plus, SpectrumOS does not have to deal with backwards compatibility. If QubesOS
> developers were to start implementing these changes, they would constantly have
> to deal with trade-offs between ease of implementing them and the cost of
> breaking backwards compatibility.
Well, that's what major version changes are for. Qubes 5 could change to a Nix model if they wanted without worrying much about backwards compatibility. I assume that won't happen for other reasons, like having higher priority work, and lack of experience with Nix.

> 
> >   At the end of the day I just don't believe that
> > trying to shoehorn these
> >  changes into Qubes is the best way to make progress.  It might well be
> >  valuable to try that, but even so it would make much more sense for
> >  somebody who believes in that approach to dedicate the huge amount of
> >  effort required to attempt it, rather than me.  This could be another
> >  effort that could be pursued in parallel to my work on Spectrum. 
> Very much this.
> 
> --
> Best,
> r

Re: Comparison to Qubes OS

[infokiller ​]

Alyssa Ross wrote:
> >       - "GUI applications are buggy, command line
> > tools
> >  are mostly
> >     undocumented": I assume that the reason for this is the lack of
> >     resources the Qubes project has. However, I don't see how this will be
> >     be better in the case of Spectrum which is a new project with one
> >     developer.   My understanding is that a lot of the instability I've
> > encountered with
> >  Qubes's tools comes down to some severe technical debt with their
> >  inter-VM communication system.  This is likely something that is very
> >  difficult to fix, but easy to learn from.  Being a new project allows
> >  Spectrum to learn from Qubes' mistakes.  I agree, but I'm still not
> > convinced this will be different in the
> >  case of Spectrum, having even more limited dev resources than
> >  Qubes'. I mean, if the Qubes folks could fix these issues without a
> >  huge effort, even if it meant rewriting all the inter-VM communication
> >  tools, they probably would. If they didn't, I assume this is because
> >  this is just a huge undertaking (as is the whole project), and they're
> >  busy with other work which has higher priority. I would assume that
> >  you will end up in exactly the same situation. 
> Maybe eventually, but the flexibility that comes with being a new
> project still gives us a significant opportunity to advance the state of
> the art before that happens.  At that point, maybe something else will
> come along to innovate free of all the technical debt Spectrum has
> acquired by then, and then the state of the art will advance even
> further.  Such is the nature of software development.
> 
> >       More generally, I'm wondering whether this
> >  projects' goals couldn't be
> >   better achieved by trying to work with the Qubes developers to
> >   integrate Nix. It may very well be that they would reject it for some
> >   reason, but then the logical next step would be to fork Qubes.  Have
> >   you reached out to the Qubes developers?   I've had the pleasure of speaking
> > with several Qubes developers on
> >  multiple occasions.  I also know that there is work being done to
> >  support NixOS templates in Qubes.    Do you have any link you can share to work on
> > Nix template being used in Qubes? 
> This was the last I heard of it:
> 
> https://hackmd.shackspace.de/qubes-nixos
> 
> >    As I see it,
> > though, the real benefit
> >  of Nix here is that it would allow defining the whole system in one
> >  single Nix configuration.  Doing this in Qubes would require big
> >  fundamental changes to how Qubes works.  I believe the idea has been
> >  brought up to the Qubes developers, and as I recall they were not keen.
> >  I believe there is room in this space for Qubes and Spectrum to coexist. 
> >  There's definitely a place for both efforts, but I'm just wondering if
> >  it would not be better to implement Spectrum's ideas while working
> >  more closely with Qubes. For example, creating a git branch or a fork
> >  that replaces Xen with KVM and integrates Nix. Now, that may very be
> >  not the best way to make progress on this project, because ramping up
> >  on the Qubes codebase will be an inefficient use of your time, or
> >  because you want to first experiment with some ideas in isolation, or
> >  for any other reason. However, the potential benefits of such a
> >  collaboration are, in my opinion, too big to pass on without strongly
> >  pursuing it. Just to name a few:
> > 
> >  - You will get feedback from top tier security experts
> >  - Qubes already has strong credibility which can be a huge driver of
> >    adoption, especially if your project will be endorsed by Qubes devs
> >  - You will be in a better position to integrate some of the tools of
> >    Qubes, which you likely would need to do, and it is less likely
> >    that changes in the Qubes tooling will break your project (if you
> >    will indeed integrate them)
> >  - You are less likely to be asked time and again "Why not Qubes"? :)
> > 
> >  So, why not start a public discussion in Qubes' mailing list on issue
> >  tracker to figure out what is needed to accomplish Spectrum's goals?
> > 
> >  It will probably turn out that you made the right decision by starting
> >  a separate project, but at the very least:
> >  - You'll may get attention from people who can contribute to Spectrum
> >  - The issues involved with be publicly documented and searchable for
> >    future generations 
> I think you maybe don't appreciate just how huge an undertaking this
> would be.  There is so much that would have to change about how Qubes
> works that I think you'd end up having to reimplement most of it
> anyway, but you'd be doing it bit by bit, never having the opportunity
> to consider the system as a whole.
> 
> At the end of the day I just don't believe that trying to shoehorn these
> changes into Qubes is the best way to make progress.  It might well be
> valuable to try that, but even so it would make much more sense for
> somebody who believes in that approach to dedicate the huge amount of
> effort required to attempt it, rather than me.  This could be another
> effort that could be pursued in parallel to my work on Spectrum.

The goal is not necessarily doing this work, but to better understand what it would require, which can guide the work on Spectrum, even if it won't share much with Qubes (since you are likely to run into some of the design decisions that Qubes had to make). Well, maybe you already figured this out. I know how Qubes works only superficially, so it may be obvious to more knowledgeable people. 
In any event, I personally would like to understand this better, as I'm interested in exploring the integration of Qubes and Nix. If you have any insights to share, it would be great, and I may also ask the Qubes devs.

Re: Comparison to Qubes OS

[Michael Raskin]

>> >   - "GUI applications are buggy, command line tools
>> > are mostly
>> >    undocumented": I assume that the reason for this is the lack of
>> >    resources the Qubes project has. However, I don't see how this will be
>> >    be better in the case of Spectrum which is a new project with one
>> >    developer. 
>> My understanding is that a lot of the instability I've encountered with
>> Qubes's tools comes down to some severe technical debt with their
>> inter-VM communication system.  This is likely something that is very
>> difficult to fix, but easy to learn from.  Being a new project allows
>> Spectrum to learn from Qubes' mistakes.
>I agree, but I'm still not convinced this will be different in the case of Spectrum, having even more limited dev resources than Qubes'. I mean, if the Qubes folks could fix these issues without a
>huge effort, even if it
>meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation.

Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes. A wl_roots based tooling is seriously considered for the first full release, after all??? A lot of problems for Qubes is that things are hard to do without making security slightly worse than what Qubes has now. Spectrum is definitely not going to achieve this level of security anytime soon for various reasons, and the goal is more to find the best trade-off between security and usability. 

There is quite a bit of design space in the gap between ??quite a bit more secure than Firejail, with the ease of use around plain NixOS plus Firejail?? and ??less secure than Qubes, but easier to manage??.

Re: Comparison to Qubes OS

[infokiller ​]

Michael Raskin wrote:
> >       - "GUI applications are buggy, command line
> > tools
> >  are mostly
> >     undocumented": I assume that the reason for this is the lack of
> >     resources the Qubes project has. However, I don't see how this will be
> >     be better in the case of Spectrum which is a new project with one
> >     developer.   My understanding is that a lot of the instability I've
> > encountered with
> >  Qubes's tools comes down to some severe technical debt with their
> >  inter-VM communication system.  This is likely something that is very
> >  difficult to fix, but easy to learn from.  Being a new project allows
> >  Spectrum to learn from Qubes' mistakes. I agree, but I'm still not
> > convinced this will be different in the case of Spectrum, having even more limited dev
> > resources than Qubes'. I mean, if the Qubes folks could fix these issues without a
> > huge effort, even if it
> > meant rewriting all the inter-VM communication tools, they probably would. If they
> > didn't, I assume this is because this is just a huge undertaking (as is the whole
> > project), and they're busy with other work which has higher priority. I would assume
> > that you will end up in exactly the same situation. 
> Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes.

True, but I'm not sure this applies to making GUI tools less buggy, or having better documentation for CLI tools. Even if Spectrum development will make big compromises, I find it unlikely that
these aspects will be improved over Qubes.
Now, I'm not saying this is not possible at all. Maybe a 100 new contributors will join the project soon and will make this viable. I just think that this is not a strong selling point of Spectrum,
which is somewhat implied in the motivation page.

> A wl_roots based tooling is seriously considered for the first full release, after all���

Is using wl_roots a non-starter for Qubes?

> A lot of problems for Qubes is that things are hard to do without making security slightly
> worse than what Qubes has now. Spectrum is definitely not going to achieve this level of
> security anytime soon for various reasons, and the goal is more to find the best trade-off
> between security and usability. 
> 
> There is quite a bit of design space in the gap between ��quite a bit more secure than
> Firejail, with the ease of use around plain NixOS plus Firejail�� and ��less secure than
> Qubes, but easier to manage��.

What do you mean by "quite a bit more secure than firejail"? isn't this side of the spectrum actually "firejail-like security"?

The way I see it, the following are the major points on the usability-security spectrum for running desktop Linux (or another desktop OS for that matter), starting from the best usability and worst
security, and ending in the worst usability and best security:

1. Run a regular Linux distro (some are better than others in providing quick security updates)
2. Harden the system: sandbox processes, harden the kernel and important userspace libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure boot, etc.
3. Use Qubes

Most users, of course, don't bother and just use (1), which is actually fine in most cases.
(2) gives pretty good usability, with the main issues related to sandboxing, since most Linux desktop apps were not built with sandboxing in mind, and the overall experience does not
support it well (for example, there's no standard permission dialogs like in Android, where sandboxing works much better in practice).
Theoretically, people could use (2), run untrusted software in VMs to get strong sandboxing but still be able to use most software, and get pretty good security. In practice this doesn't quite work
because running stuff in dedicated VMs is not streamlined enough, so people just don't do it.
Then there's (3), which is the most secure, but has some usability issues that were already mentioned. So Spectrum can fill the gap between (2) and (3).

Re: Comparison to Qubes OS

[Michael Raskin]

>> > resources than Qubes'. I mean, if the Qubes folks could fix these issues without a
>> > huge effort, even if it
>> > meant rewriting all the inter-VM communication tools, they probably would. If they
>> > didn't, I assume this is because this is just a huge undertaking (as is the whole
>> > project), and they're busy with other work which has higher priority. I would assume
>> > that you will end up in exactly the same situation. 
>> Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes.
>
>True, but I'm not sure this applies to making GUI tools less buggy, or having better documentation for CLI tools. 

Yes, it does apply. SpectrumOS is willing to accept quite a bit more of external code with good in-the-wild track record inside the trusted code base,
at least for _most_ tasks. So where Qubes needs to reimplement something with a completely different design (allowing minimisation of TCB), Spectrum
can (at least in the beginning, sometimes grudgingly) take the closest thing in existence and add only critically missing parts.

>> A wl_roots based tooling is seriously considered for the first full release, after all?????????
>
>Is using wl_roots a non-starter for Qubes?

Yes, and I hope it is a complete non-starter on the level of not being worth any discussion there.

I can only remind you, that Qubes has doubts about KVM, a widely used part of the Linux kernel, being sufficiently secure. Thety have the same stance
about half of the features of Xen, the hypervisor they do use.  This does make some things Spectrum plans to do for usability and performance much
harder to implement, as the underlying features are undesirable in the TCB.

Compared to this, wl_roots is _way_ more dangerous. This is a library which has segfaults in relatively normal use, and apparently the users do hit
these relatively frequently. Some of these seem to survive for months. I have not looked whether any of these are exploitable, but for the Qubes level
of requirements one should just assume it is. Notice that for Spectrum one could also reuse some kind of VNC code if wl_roots is eventually considered
too buggy. For Qubes even VNC libraries are far from good enough.

>> There is quite a bit of design space in the gap between ??????quite a bit more secure than
>> Firejail, with the ease of use around plain NixOS plus Firejail?????? and ??????less secure than
>> Qubes, but easier to manage??????.
>What do you mean by "quite a bit more secure than firejail"? isn't this side of the spectrum actually "firejail-like security"?

Firejail depends on namespaces, which still have some weird behaviours in some corner cases, there is a hope that VMs will be a simpler foundation.

>The way I see it, the following are the major points on the usability-security spectrum for running desktop Linux (or another desktop OS for that matter), starting from the best usability and worst
>security, and ending in the worst usability and best security:
>
>1. Run a regular Linux distro (some are better than others in providing quick security updates)
>2. Harden the system: sandbox processes, harden the kernel and important userspace libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure boot, etc.

If you do not understand what you are doing well enough, Wayland as you use it might end up being less secure than X11, by the way.

>Most users, of course, don't bother and just use (1), which is actually fine in most cases.
>(2) gives pretty good usability, with the main issues related to sandboxing, since most Linux desktop apps were not built with sandboxing in mind, and the overall experience does not
>support it well (for example, there's no standard permission dialogs like in Android, where sandboxing works much better in practice).

Actually, if you write a few relatively reasonable wrappers around some kind of namespace-based sandboxing, usability problems kind of become quite
different from the normal ones (some UI flows are actually better when the choices are trimmed in advance??? and also suddenly a lot of things do not
need to be chosen uniformly at the entire-system level anymore). No idea how much effort is to make meaningful GUI permission dialogs. Android ones
are clearly not meaningful enough, of course.

Re: Comparison to Qubes OS

[infokiller ​]

Michael Raskin wrote:
> >     resources than Qubes'. I mean, if the Qubes folks
> > could fix these issues without a
> >  huge effort, even if it
> >  meant rewriting all the inter-VM communication tools, they probably would. If they
> >  didn't, I assume this is because this is just a huge undertaking (as is the whole
> >  project), and they're busy with other work which has higher priority. I would assume
> >  that you will end up in exactly the same situation.   Note that SpectrumOS is going
> > to make tradeoffs that are complete non-starters for Qubes. 
> > True, but I'm not sure this applies to making GUI tools less buggy, or having better
> > documentation for CLI tools.  
> Yes, it does apply. SpectrumOS is willing to accept quite a bit more of external code with
> good in-the-wild track record inside the trusted code base,
> at least for _most_ tasks. So where Qubes needs to reimplement something with a completely
> different design (allowing minimisation of TCB), Spectrum
> can (at least in the beginning, sometimes grudgingly) take the closest thing in existence
> and add only critically missing parts.
I disagree. Spectrum still needs to write its own GUI tools. Since the Qubes CLI tools are not buggy, presumably the bugs Alyssa ran into are not related to the Qubes design, but rather to GUI programming. As for the CLI tools, I'm really not sure if she can actually reuse any docs, or write less of them. AFAIK, most of the tools in question are related to inter-VM comms, which are not just out there in a fully documented form.
And even if it were the case that it would be possible to significantly reuse existing tools, it's still speculative at best to claim that this will overcome the clear disadvantage in development resources.
> 
> >    A wl_roots
> > based tooling is seriously considered for the first full release, after all���������
> > 
> > Is using wl_roots a non-starter for Qubes? 
> Yes, and I hope it is a complete non-starter on the level of not being worth any
> discussion there.
> 
> I can only remind you, that Qubes has doubts about KVM, a widely used part of the Linux
> kernel, being sufficiently secure. Thety have the same stance
> about half of the features of Xen, the hypervisor they do use.  This does make some things
> Spectrum plans to do for usability and performance much
> harder to implement, as the underlying features are undesirable in the TCB.
> 
> Compared to this, wl_roots is _way_ more dangerous. This is a library which has segfaults
> in relatively normal use, and apparently the users do hit
> these relatively frequently. Some of these seem to survive for months. I have not looked
> whether any of these are exploitable, but for the Qubes level
> of requirements one should just assume it is. Notice that for Spectrum one could also
> reuse some kind of VNC code if wl_roots is eventually considered
> too buggy. For Qubes even VNC libraries are far from good enough.
Well, Qubes does intend to experiment with Wayland [1], though possibly they could use smithay [2] or build their own, hopefully in a memory safe language.
And there's also work in Qubes on moving the GUI part from dom0, which will lower the risk related to the GUI.

[1] https://www.qubes-os.org/gsoc/#wayland-support-in-gui-agent-andor-gui-daemon
[2] https://github.com/Smithay/smithay
> 
> >    There is quite
> > a bit of design space in the gap between ������quite a bit more secure than
> >  Firejail, with the ease of use around plain NixOS plus Firejail������ and ������less
> > secure than
> >  Qubes, but easier to manage������. What do you mean by "quite a bit more
> > secure than firejail"? isn't this side of the spectrum actually
> > "firejail-like security"? 
> Firejail depends on namespaces, which still have some weird behaviours in some corner
> cases, there is a hope that VMs will be a simpler foundation.
I understand, but it sounded like this side of the spectrum should be "using NixOs, maybe with Firejail", in which case the security is Firejail-like.
> 
> >  The way I see it, the following are the major points on
> > the usability-security spectrum for running desktop Linux (or another desktop OS for that
> > matter), starting from the best usability and worst
> > security, and ending in the worst usability and best security:
> > 
> > 1. Run a regular Linux distro (some are better than others in providing quick security
> > updates)
> > 2. Harden the system: sandbox processes, harden the kernel and important userspace
> > libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure
> > boot, etc. 
> If you do not understand what you are doing well enough, Wayland as you use it might end
> up being less secure than X11, by the way.
What do you mean?
> 
> >  Most users, of course, don't bother and just use
> > (1), which is actually fine in most cases.
> > (2) gives pretty good usability, with the main issues related to sandboxing, since most
> > Linux desktop apps were not built with sandboxing in mind, and the overall experience does
> > not
> > support it well (for example, there's no standard permission dialogs like in Android,
> > where sandboxing works much better in practice). 
> Actually, if you write a few relatively reasonable wrappers around some kind of
> namespace-based sandboxing, usability problems kind of become quite
> different from the normal ones (some UI flows are actually better when the choices are
> trimmed in advance��� and also suddenly a lot of things do not
> need to be chosen uniformly at the entire-system level anymore). 
I'm curious about these wrappers, could you elaborate?
> No idea how much effort
> is to make meaningful GUI permission dialogs. Android ones
> are clearly not meaningful enough, of course.
Android's permission system is far from perfect, but it also targets a very large userbase that doesn't care much about security (as is Windows). And it's still **much** better than desktop Linux.

Re: Comparison to Qubes OS

[Michał "rysiek" Woźniak]

[-- Attachment #1.1: Type: text/plain, Size: 2223 bytes --]

Hey,

On 6/15/20 1:44 PM, infokiller wrote:
> Michael Raskin wrote:
>>>     resources than Qubes'. I mean, if the Qubes folks
>>> could fix these issues without a
>>>  huge effort, even if it
>>>  meant rewriting all the inter-VM communication tools, they probably would. If they
>>>  didn't, I assume this is because this is just a huge undertaking (as is the whole
>>>  project), and they're busy with other work which has higher priority. I would assume
>>>  that you will end up in exactly the same situation.   Note that SpectrumOS is going
>>> to make tradeoffs that are complete non-starters for Qubes. 
>>> True, but I'm not sure this applies to making GUI tools less buggy, or having better
>>> documentation for CLI tools.  
>> Yes, it does apply. SpectrumOS is willing to accept quite a bit more of external code with
>> good in-the-wild track record inside the trusted code base,
>> at least for _most_ tasks. So where Qubes needs to reimplement something with a completely
>> different design (allowing minimisation of TCB), Spectrum
>> can (at least in the beginning, sometimes grudgingly) take the closest thing in existence
>> and add only critically missing parts.
> I disagree.

And you have all the right to.

However, it seems to me that the reasons behind SpectrumOS being done the way
it's done have been pretty extensively explained and documented in this thread.
At this point the discussion seems to focus on back-of-a-napkin estimates
regarding what is a more effective way of implementing certain things.

Clearly your estimates are different than those of others responding in this
thread. You might be right and we might be wrong, but the only way to test that
is to start implementing things one way or the other.

SpectrumOS made their estimations and arrived at a decision to do things one
particular way. You may disagree this is the best way to do things, but I feel
at this point a better way of making your point would be for you to start
implementing things the way you believe is more effective.

SpectrumOS is basically a one-person team. If you're right, you should be able
to overtake SpectrumOS development pretty quickly.

--
Best,
r


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Comparison to Qubes OS

[infokiller ​]

Michał "rysiek" Woźniak wrote:
> Hey,
> 
> On 6/15/20 1:44 PM, infokiller wrote:
> >   Michael Raskin wrote:
> >       
> > resources than Qubes'. I mean, if the Qubes folks
> >  could fix these issues without a
> >   huge effort, even if it
> >   meant rewriting all the inter-VM communication tools, they probably would. If they
> >   didn't, I assume this is because this is just a huge undertaking (as is the whole
> >   project), and they're busy with other work which has higher priority. I would
> > assume
> >   that you will end up in exactly the same situation.   Note that SpectrumOS is going
> >  to make tradeoffs that are complete non-starters for Qubes. 
> >  True, but I'm not sure this applies to making GUI tools less buggy, or having better
> >  documentation for CLI tools.    Yes, it does apply. SpectrumOS is willing to accept
> > quite a bit more of external code with
> >  good in-the-wild track record inside the trusted code base,
> >  at least for _most_ tasks. So where Qubes needs to reimplement something with a
> > completely
> >  different design (allowing minimisation of TCB), Spectrum
> >  can (at least in the beginning, sometimes grudgingly) take the closest thing in
> > existence
> >  and add only critically missing parts.  I disagree. 
> And you have all the right to.
> 
> However, it seems to me that the reasons behind SpectrumOS being done the way
> it's done have been pretty extensively explained and documented in this thread.
> At this point the discussion seems to focus on back-of-a-napkin estimates
> regarding what is a more effective way of implementing certain things.
I was mainly responding to understand if I'm missing anything.
> 
> Clearly your estimates are different than those of others responding in this
> thread. You might be right and we might be wrong, but the only way to test that
> is to start implementing things one way or the other.
> 
> SpectrumOS made their estimations and arrived at a decision to do things one
> particular way. You may disagree this is the best way to do things, but I feel
> at this point a better way of making your point would be for you to start
> implementing things the way you believe is more effective.

I'm not trying to make any point, and I honestly don't have any stakes here. I'm trying to figure out why things are done the way they are.
My ultimate goal is to have a secure, performant, and easy to manage OS. And so, I would be very happy if SpectrumOS will be able to satisfy these requirements.

> 
> SpectrumOS is basically a one-person team. If you're right, you should be able
> to overtake SpectrumOS development pretty quickly.

I'm not saying integrating with Qubes is quick or easy: I'm arguing that it may be the better approach to reach the project goals in a sustainable way. 
Either way, I won't have much time to work on something like this anytime soon. That said, though I'd like to figure out what it involves, and if it's reasonable to expect I'll be able to make incremental progress as a side project.

> 
> --
> Best,
> r

Re: Comparison to Qubes OS

[Michael Raskin]

>I disagree. Spectrum still needs to write its own GUI tools. 

There are some XDG tools that can be reused wholesale.

>> >  Qubes, but easier to manage??????????????????. What do you mean by "quite a bit more
>> > secure than firejail"? isn't this side of the spectrum actually
>> > "firejail-like security"? 
>> Firejail depends on namespaces, which still have some weird behaviours in some corner
>> cases, there is a hope that VMs will be a simpler foundation.
>I understand, but it sounded like this side of the spectrum should be "using NixOs, maybe with Firejail", in which case the security is Firejail-like.

SpectrumOS will use small VMs, not containers.

>> > 2. Harden the system: sandbox processes, harden the kernel and important userspace
>> > libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure
>> > boot, etc. 
>> If you do not understand what you are doing well enough, Wayland as you use it might end
>> up being less secure than X11, by the way.
>What do you mean?

Well, most compositors allow clients quite a lot of things that theoretically should be 
permission-checked, so there is not as much good to overcome the drawbacks of a lot of code
being still a bit raw.

>> >  Most users, of course, don't bother and just use
>> > (1), which is actually fine in most cases.
>> > (2) gives pretty good usability, with the main issues related to sandboxing, since most
>> > Linux desktop apps were not built with sandboxing in mind, and the overall experience does
>> > not
>> > support it well (for example, there's no standard permission dialogs like in Android,
>> > where sandboxing works much better in practice). 
>> Actually, if you write a few relatively reasonable wrappers around some kind of
>> namespace-based sandboxing, usability problems kind of become quite
>> different from the normal ones (some UI flows are actually better when the choices are
>> trimmed in advance????????? and also suddenly a lot of things do not
>> need to be chosen uniformly at the entire-system level anymore). 
>I'm curious about these wrappers, could you elaborate?

Well, I run browser instances and PDF/image/video viewers inside nsjail, and the wrappers
_mainly_ just prepare a reasonably restricted access to give the program running inside 
nsjail. Of course if I bother to do all that, there is also use of single-use UIDs on top
of restrictions by mount namespaces.

It's not that much code, and in my case it is in Common Lisp.

And then it is just all the boring stuff that is annoying elsewhere like using different
resolv.conf for different applications and giving applications access only through proxy
and only picking from the files that the application should use and not navigating the
entire storage.

So I watch SpectrumOS mostly looking at the CrosVM work and when it will be sufficient 
for replacing containers and socket proxying with VMs and virtio-based socket proxying.

>> No idea how much effort
>> is to make meaningful GUI permission dialogs. Android ones
>> are clearly not meaningful enough, of course.
>Android's permission system is far from perfect, but it also targets a very large userbase that doesn't care much about security (as is Windows). And it's still **much** better than desktop Linux.

It is unclear that Android in practice (not how the intents should have been used) is better than
desktop Linux with Flatpack everywhere, let alone the unfortunately rare case of desktop Linux
where people _also_ use UID segregation. There is a chance that XDG-portal stuff ends up closer to
how intents should have been used than what we currently see on Android. And recently it looks like
Android is now going to give up the remaining appearances of being usable as a general-purpose
computing platform, but this seems not to be forced by the permission system design in any way.

Re: Comparison to Qubes OS

[infokiller ​]

Michael Raskin wrote:
> >  I disagree. Spectrum still needs to write its own GUI
> > tools.  
> There are some XDG tools that can be reused wholesale.
> 
> >      Qubes, but easier to manage������������������. What
> > do you mean by "quite a bit more
> >  secure than firejail"? isn't this side of the spectrum actually
> >  "firejail-like security"?   Firejail depends on namespaces, which still
> > have some weird behaviours in some corner
> >  cases, there is a hope that VMs will be a simpler foundation. I understand, but it
> > sounded like this side of the spectrum should be "using NixOs, maybe with
> > Firejail", in which case the security is Firejail-like. 
> SpectrumOS will use small VMs, not containers.

I understand that: I was referring to a hypothetical system running NixOS with Firejail, which is lower security and higher usability than Spectrum/Qubes.

> 
> >     2. Harden the system: sandbox processes, harden the
> > kernel and important userspace
> >  libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure
> >  boot, etc.   If you do not understand what you are doing well enough, Wayland as
> > you use it might end
> >  up being less secure than X11, by the way. What do you mean? 
> Well, most compositors allow clients quite a lot of things that theoretically should be 
> permission-checked, so there is not as much good to overcome the drawbacks of a lot of
> code
> being still a bit raw.
> 
> >      Most users, of course, don't bother and just use
> >  (1), which is actually fine in most cases.
> >  (2) gives pretty good usability, with the main issues related to sandboxing, since most
> >  Linux desktop apps were not built with sandboxing in mind, and the overall experience
> > does
> >  not
> >  support it well (for example, there's no standard permission dialogs like in
> > Android,
> >  where sandboxing works much better in practice).   Actually, if you write a few
> > relatively reasonable wrappers around some kind of
> >  namespace-based sandboxing, usability problems kind of become quite
> >  different from the normal ones (some UI flows are actually better when the choices are
> >  trimmed in advance��������� and also suddenly a lot of things do not
> >  need to be chosen uniformly at the entire-system level anymore).  I'm curious
> > about these wrappers, could you elaborate? 
> Well, I run browser instances and PDF/image/video viewers inside nsjail, and the wrappers
> _mainly_ just prepare a reasonably restricted access to give the program running inside 
> nsjail. Of course if I bother to do all that, there is also use of single-use UIDs on top
> of restrictions by mount namespaces.
> 
> It's not that much code, and in my case it is in Common Lisp.
> 
> And then it is just all the boring stuff that is annoying elsewhere like using different
> resolv.conf for different applications and giving applications access only through proxy
> and only picking from the files that the application should use and not navigating the
> entire storage.
Yeah, I find this is indeed the most annoying stuff when I'm using Firejail. It's especially annoying if I realize I want to open a file in the sandboxed app that I didn't whitelist, which means I rerun it (there may be a way to whitelist files after the app is already running, but I never bothered to check).
> 
> So I watch SpectrumOS mostly looking at the CrosVM work and when it will be sufficient 
> for replacing containers and socket proxying with VMs and virtio-based socket proxying.
> 
> >    No idea how
> > much effort
> >  is to make meaningful GUI permission dialogs. Android ones
> >  are clearly not meaningful enough, of course. Android's permission system is
> > far from perfect, but it also targets a very large userbase that doesn't care much
> > about security (as is Windows). And it's still **much** better than desktop Linux.
> > 
> It is unclear that Android in practice (not how the intents should have been used) is
> better than
> desktop Linux with Flatpack everywhere, let alone the unfortunately rare case of desktop
> Linux
> where people _also_ use UID segregation. There is a chance that XDG-portal stuff ends up
> closer to
> how intents should have been used than what we currently see on Android. And recently it
> looks like
> Android is now going to give up the remaining appearances of being usable as a
> general-purpose
> computing platform, but this seems not to be forced by the permission system design in any
> way.
I think Flatpak is a step in the right direction, but is still far from being everywhere, and has its own share of issues. It also doesn't solve the issue of apps that were not built with a permission system.
Android has other issues, of course, like very slow (it at all) security updates, but if you're using a recent Pixel you are probably much better off than using desktop Linux.