Overview

Magic Nix VFS is a virtual filesystem implemented using FUSE that provides /nix/store instead of an actual disk partition. It is backed by NAR archives on disk, and makes the files packaged in these archives visible to users as regular files which can be used directly.

Magic Nix VFS can automatically download and serve NAR archives for any paths found in configured cache servers. For instance, if you enable the cache.nixos.org cache server, then all programs built by the NixOS project will be directly visible in /nix/store and can be launched immediately, without an invocation of the Nix toolchain to download the closure.

Magic Nix VFS thus proposes a new trade-off where we pay for some RAM usage and upfront network traffic, and in return we optimize certain scenarios and unlock new use cases for the Nix/NixOS ecosystem.

Example scenarios

Demo scenario: net booting from a public cache server

This scenario is currently live, and you can run it yourself! See instructions in the usage chapter.

How it works: A public Nix cache is set up at https://cache.magicnix.adnab.me that contains two pre-build NixOS systems, one that runs a full Cinnamon GUI and one that boots to CLI.

A minimal ISO is provided that allows booting into these systems by mounting /nix/store as a magic Nix VFS. Files necessary to boot and run the system will be downloaded lazily by Magic Nix VFS as they are used. The ISO image itself contains only a kernel and an initrd, and is only 28MB.

Note that the two variants (Cinnamon GUI and CLI-only) use the same kernel and initrd, the only difference is in the kernel parameter init=, which names the Nix store path of the NixOS system we want to boot into. Thus, any number of NixOS variants could be booted from this ISO at no cost.

Scenario 2: HPC cluster

An HPC cluster is configured using NixOS. A head node contains the Nix store, and all compute nodes want to access paths from this store directly in /nix/store.

With current tools, a common solution would be to mount /nix/store from the head node using NFS. This incurs a significant penalty especially when reading many small files, for instance when doing an evaluation of Nixpkgs (several thousands of .nix files).

With Magic Nix VFS, the head node can run a Nix cache to be used by the compute nodes. Instead of requesting files one by one to the NFS server, incurring a round-trip-time latency each time, each time a store path is accessed, the entire path would be prefetched from the head node by Magic Nix VFS, and subsequent access to files inside the path would be local and thus very fast.

To enable the full Nix toolchain to be used on compute nodes, all builds can be delegated to the head node using the experimental mounted-ssh-store feature of Nix, or simply by proxying the Nix daemon socket to the head node via a TCP connection. In the future, Magic Nix VFS will also support mounting /nix/store read-write so that builds can run on any compute node and then be copied back to the head node.

Scenario 3: server cluster made of old machines

This scenario is similar to the infrastructure setup used by Deuxfleurs to host their services. A cluster of servers is built with NixOS with the intent of providing a variety of web services (websites, email, …). The machines are mostly old machines that are quite slow. In particular, running Nix builds takes a long time, so we want to avoid doing it from the servers if possible.

The cluster runs an orchestrator that automatically runs services on the various nodes. These services run from the Nix store, and need to start fast, as the orchestrator might need to move services from one node to another at any time (for example if a node goes down) and we want to minimize downtime when doing so.

The cluster runs, among other things, a distributed storage service such as Garage which can be used to serve a Nix cache.

Without Magic Nix VFS, running this setup would be quite cumbersome, with basically two options:

• Include the full Nix build recipe in the service definition of each service, so that when it starts the Nix build is ran to ensure that the necessary artifacts are built before launching the service. This incurs a big penalty in terms of time needed to start services.

  The build time can be reduced by pre-building these artifacts and adding them to a cache server, but the runner still needs to evaluate the whole Nix build recipe whenever starting the service to determine which paths are needed. Nix evaluation times can be quite long especially on underpowered machines, due to the size of the Nixpkgs repository.

• Whenever a service is created or modified, build the artifacts required to run this service on a local machine and then copy these artifacts to all nodes using nix-copy. The service definition in the orchestrator can then simply be to run a path from /nix/store.

  This leads to optimal startup time as there is no build/fetch step when running a job, but this makes the deployment of a new service longer due to the copy step which is multiplied by the number of nodes. This would be especially painful during development when rapid iteration on the service definition is required.

  Note that in this scenario, the artifacts MUST be locally present on all nodes before the service definition is added to the orchestrator, otherwise the service might not be able to start.

With Magic Nix VFS, we get a new combination that mostly solves these issues:

• Build services locally on a more powerful machine, and upload them to a single location: the Nix cache server.

• The service definition does not include a Nix build step, it runs the service directly from a binary in /nix/store named by its final path.

• When a service is started on a node, Magic Nix VFS might need to fetch some artifacts from the cache server. Crucially, knowing which artifacts to download does not require a Nix evaluation, so this process will only take the time strictly required to fetch the necessary files from the cache. If the node had already downloaded these artifacts previously, no download is required and the service will start immediately.