[spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices

[Michael Raskin <7c6f434c@mail.ru>]

>>>+State directories and applications will be <var>m</var>:<var>n</var>.
>>
>> Note (probably not affecting the design right now): this can also be
>> a source of annoyance with running everything under unique single-use
>> UIDs: to have multiple UIDs access the same resource, setting ACLs is
>> a natural idea. It works well, but might require some cleanup of 
>> obsolete ACLs for a workflow with a large churn of instances ??? the 
>> maximum length of an ACL is limited.
>
>Yeah, we'll need something like that.  ACLs sound sensible.

Of course, this is a persistent state related to a large number of 
inodes that requires keeping track of used unique UIDs (to avoid reuse
before cleanup) and cleanup (I have learned it hte hard way: one does 
run out of ACL capacity without removing the obsolete ones), including
some kind of cleanup in case of VM crash or unclean host reboot.

>> Given that it is sometimes convenient to spawn multiple VMs with almost 
>> identical settings on the fly (e.g. Firefox instances but with different
>> state directories for downloads/uploads), should we commit to define
>> a CLI layer for starting VMs (probably similar to the underlying VMM
>> software CLI, but adapted to the notions we want to use), with the 
>> Nix-based management of VM fleet built on top of this CLI?
>
>Yeah, I've been thinking about this too and agree.  We should have a nix
>run analogue for temporary VMs.  It would be nice for scripts that use
>multiple VMs to be able to start them on the fly as well.  (But we
>should definitely not have a nix-env -i analogue, because there should
>be no hidden state on a host.)>

I think it is better to avoid evaluation, actually.

So I want a level where there is a Spectrum Profile Generation with 
named prebuilt VMs complexity levels (console-only, for stuff like 
ImageMagick on images downloaded from 4chan; GUI-but-no-HW, to remove 
GPU attack surface and guarantee lack of sound; full-GUI, which is still
not guaranteed to get access to everything, of course), and prebuilt 
application sets (just buildEnv stuff, no fancy stuff).

Then I say:

spectrum-vm --base full-gui --net nat --app firefox-esr-js-profile \
--app gvim --dri card0 --rw-directory user:conf/nixcon-2020 \
-- firefox https://nixos.org/

This should not do anything in run time that can be safely cached. VM is
prebuilt, application bunches are prebuilt and just mounted into store,
even Firefox profile is prebuilt and changes are going to be discarded.
No builds, no Nix evaluations.

(yes, there is also the annoying question of RAM amount management, not
sure how to do it well, ballooning would be the nicest thing if it is 
feasible)

>> I suspect that building things the other way round is not much simpler,
>> but building a single VM image usable with slightly different settings
>> will have smaller overhead than evaluating a Nix expression for each
>> extra VM.
>
>Maybe.  I'll have to think more about that.  tbh I think we can assume
>that the evaluator will get faster at some point (because there's big
>potential for optimisations and caching in safe mode).  It might be nice

Oh well, in the safe mode there is even a potential for precompiling Nix
functions. It's another question whether we want to experiment with
workflows without having access to such speedups.

And even a pure evaluation of a checkout needs to stat all the relevant
files (even if we accept the risk of not checksumming them).

>to invent yet another way to do options, and just keep that in Nix.  But
>I could be swayed either way.

I assume there are good arguments for the config to be structured (and
in my current system there are some interface boundaries with structured
config and some wrapper scripts that manage to generate almost all the 
usecases using command-line arguments, but sometimes parts of the 
structure need to be passed to them).

On the other hand, JSON can be parsed and written easily in all
languages we might want to use, even in Bash using jq. And we need some
code to setup the namespaces, but nsjail/firejail/bubblewrap are C code
so in the long term there would be some rewrite, probably in Rust??? And
having Nix preprocess arguments for Rust code sounds strange.

And whatever you do, Nix evaluation is always just another layer before
running the actual code for setting up the isolated environments that 
still needs to interpret its arguments.

Also, I might want to use some binary cache, but when I am offline Nix
builds wait for a long time for a reply from a binary cache. It's not so
bad if I am intentionally building something and can pass an empty value
for binary-caches via the command line, but doing it for each new 
command I execute sounds excessive.

Also, it might be convenient to pass entire Nix store inside some VMs,
but if all the _environments_ of all the currently running VMs are 
visible in the Nix store??? I really shouldn't.

I guess if I mention multiuser systems you would say that we need to
allow chmod a-x on /nix/store?