Overview

Magic Nix VFS is a virtual filesystem implemented using FUSE that provides /nix/store instead of an actual disk partition. It is backed by NAR archives on disk, and makes the files packaged in these archives visible to users as regular files which can be used directly.

Magic Nix VFS can automatically download and serve NAR archives for any paths found in configured cache servers. For instance, if you enable the cache.nixos.org cache server, then all programs built by the NixOS project will be directly visible in /nix/store and can be launched immediately, without an invocation of the Nix toolchain to download the closure.

Magic Nix VFS thus proposes a new trade-off where we pay for some RAM usage and upfront network traffic, and in return we optimize certain scenarios and unlock new use cases for the Nix/NixOS ecosystem.

Example scenarios

Demo scenario: net booting from a public cache server

This scenario is currently live, and you can run it yourself! See instructions in the usage chapter.

How it works: A public Nix cache is set up at https://cache.magicnix.adnab.me that contains two pre-build NixOS systems, one that runs a full Cinnamon GUI and one that boots to CLI.

A minimal ISO is provided that allows booting into these systems by mounting /nix/store as a magic Nix VFS. Files necessary to boot and run the system will be downloaded lazily by Magic Nix VFS as they are used. The ISO image itself contains only a kernel and an initrd, and is only 28MB.

Note that the two variants (Cinnamon GUI and CLI-only) use the same kernel and initrd, the only difference is in the kernel parameter init=, which names the Nix store path of the NixOS system we want to boot into. Thus, any number of NixOS variants could be booted from this ISO at no cost.

Scenario 2: HPC cluster

An HPC cluster is configured using NixOS. A head node contains the Nix store, and all compute nodes want to access paths from this store directly in /nix/store.

With current tools, a common solution would be to mount /nix/store from the head node using NFS. This incurs a significant penalty especially when reading many small files, for instance when doing an evaluation of Nixpkgs (several thousands of .nix files).

With Magic Nix VFS, the head node can run a Nix cache to be used by the compute nodes. Instead of requesting files one by one to the NFS server, incurring a round-trip-time latency each time, each time a store path is accessed, the entire path would be prefetched from the head node by Magic Nix VFS, and subsequent access to files inside the path would be local and thus very fast.

To enable the full Nix toolchain to be used on compute nodes, all builds can be delegated to the head node using the experimental mounted-ssh-store feature of Nix, or simply by proxying the Nix daemon socket to the head node via a TCP connection. In the future, Magic Nix VFS will also support mounting /nix/store read-write so that builds can run on any compute node and then be copied back to the head node.

Scenario 3: server cluster made of old machines

This scenario is similar to the infrastructure setup used by Deuxfleurs to host their services. A cluster of servers is built with NixOS with the intent of providing a variety of web services (websites, email, …). The machines are mostly old machines that are quite slow. In particular, running Nix builds takes a long time, so we want to avoid doing it from the servers if possible.

The cluster runs an orchestrator that automatically runs services on the various nodes. These services run from the Nix store, and need to start fast, as the orchestrator might need to move services from one node to another at any time (for example if a node goes down) and we want to minimize downtime when doing so.

The cluster runs, among other things, a distributed storage service such as Garage which can be used to serve a Nix cache.

Without Magic Nix VFS, running this setup would be quite cumbersome, with basically two options:

• Include the full Nix build recipe in the service definition of each service, so that when it starts the Nix build is ran to ensure that the necessary artifacts are built before launching the service. This incurs a big penalty in terms of time needed to start services.

  The build time can be reduced by pre-building these artifacts and adding them to a cache server, but the runner still needs to evaluate the whole Nix build recipe whenever starting the service to determine which paths are needed. Nix evaluation times can be quite long especially on underpowered machines, due to the size of the Nixpkgs repository.

• Whenever a service is created or modified, build the artifacts required to run this service on a local machine and then copy these artifacts to all nodes using nix-copy. The service definition in the orchestrator can then simply be to run a path from /nix/store.

  This leads to optimal startup time as there is no build/fetch step when running a job, but this makes the deployment of a new service longer due to the copy step which is multiplied by the number of nodes. This would be especially painful during development when rapid iteration on the service definition is required.

  Note that in this scenario, the artifacts MUST be locally present on all nodes before the service definition is added to the orchestrator, otherwise the service might not be able to start.

With Magic Nix VFS, we get a new combination that mostly solves these issues:

• Build services locally on a more powerful machine, and upload them to a single location: the Nix cache server.

• The service definition does not include a Nix build step, it runs the service directly from a binary in /nix/store named by its final path.

• When a service is started on a node, Magic Nix VFS might need to fetch some artifacts from the cache server. Crucially, knowing which artifacts to download does not require a Nix evaluation, so this process will only take the time strictly required to fetch the necessary files from the cache. If the node had already downloaded these artifacts previously, no download is required and the service will start immediately.

Usage

Magic Nix VFS demo

Requirements:

• a computer or VM
• with a wired Internet connection with automatic DHCP configuration
• and at least 8GB of RAM (for the destkop variant) or 2GB (for the CLI variant)

Download:

• magic-nixos-demo-v1.iso (28MB)

Instructions:

1. Download the ISO file above.

2. If booting a physical computer, burn ISO to a CD-ROM or USB pendrive.

3. Boot computer or VM from this disk.

   Here is an example command to run within a Qemu virtual machine:

   qemu-system-x86_64 -m 8G -enable-kvm -smp 2 -cdrom magic-nixos-demo-v1.iso
   

4. Select the variant of the demo to run: full desktop with Cinnamon, or CLI-only. The CLI variant will boot faster but has less functionality.

5. If using the CLI variant, login with username root and password root.

6. Explore the system.

Running programs:

If you are running the desktop version of the demo, note that no extra programs are installed by default. However, all binary packages ever built by the NixOS project and present on cache.nixos.org can be run simply by launching their full path. For instance, try launching one of the following commands:

• /nix/store/kq5pl1xnpzmkh1x6dzy857l0929bkgx5-thunderbird-152.0.1/bin/thunderbird
• /nix/store/4b7ydwmf41lzcha054wf4m5nki4fb2ml-inkscape-1.4.4/bin/inkscape

Using the magicnixvfs NixOS module

You can use Magic Nix VFS on your own system thanks to the NixOS module provided in this repository, but consider such a setup unstable and use it only for testing purposes.

To do so, you must build your system configuration as a flake. Here is an example flake.nix for doing so:

{
  description = "An example system configuration that imports the Magic Nix VFS module";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
    magicnixvfs.url = "git+https://git.deuxfleurs.fr/lx/magic-nix-vfs.git";
  };

  outputs = { self, nixpkgs, magicnixvfs, ...}: {
    nixosConfigurations."example" = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        magicnixvfs.nixosModules."magicnixvfs"
        ./configuration.nix
      ];
    };
  };
}

For usage examples, you can read the configuration.nix file from the netboot demo. The options for configuring the magicnixvfs module are listed on the configuration reference page.

Note that this works for netbooting when the NixOS configuration is built on another machine and uploaded to a cache server. If boot.magicnixvfs.statePath (default: /var/lib/magicnixvfs) is mounted on a local disk, Magic Nix VFS will download and store there all files necessary to boot and run the system when it first boots. Future boots will then be much faster, with all files already available locally.

Note that since Magic Nix VFS does not yet have write support, it is not yet possible to build the NixOS configuration on the same machine that will run it. This also means that using the standard NixOS installation procedure is not possible.

Internals

Inside Magic Nix VFS

High-level overview

Magic Nix VFS implements a FUSE-based filesystem that is mounted under /nix/store. It is divided in two main parts:

• A main thread that runs the FUSE server. This thread keeps metadata stored in RAM and tries to answer metadata requests on store files as fast as possible

• A multi-threaded I/O component based on a thread pool. The pool runs I/O jobs in the background, such as loading NAR archives from disk or downloading them from a cache server as necessary.

In practice, the thread pool is actually divided in two separate pools for performance reasons.

• A “small task” pool that answers read requests from NARs already on disk.

• A “big task” pool that loads new NARs, downloading them from the network if necessary.

Read path

The main thread keeps the following metadata in RAM:

• For recently accessed and currently referenced NAR archives, the list of all files in the archive (and their associated inode number) as well as the position of their actual content in the archive

• Information about paths that are currently being loaded by a background thread (from disk or downloaded from the network)

All of this information is NOT guarded by a mutex lock, it is fully local to this thread to enable answering FUSE requests as fast as possible.

When reading data from a Nix path which is served from a NAR archive, several code paths are possible:

• For metadata requests such as stat or readdir on paths that already have their metadata loaded in RAM, the single threaded part can answer immediately.

• For the read system call which is used to read actual data from files within the NAR archives, the main thread looks up in metadata the corresponding offset to read from, and then launches a new “small task” on the thread pool to do the actual read. The main thread keeps on answering other requests in the meantime.

  The task delegated to the thread pool consist simply of doing a read syscall on the NAR file at the given offset, and then replying to the FUSE request with the read data. The read request does not pass back through the main thread, the worker thread can answer it directly.

Dynamically loading new NARs

When the main thread recieves a metadata request on a path that is not yet loaded, a “big task” is pushed to the thread pool to retrieve the path and load its metadata in RAM. During the time when the store path is being loaded, all requests for this path are transferred to the offloaded task, so that the main thread can continue processing requests for other paths.

The offloaded tasks runs the following steps:

• Check whether the path is already present on disk

• If it isn’t, find the path on a cache server and download it

• Load the narinfo file, check it is consistent, and check the signature matches the content of the NAR file

• Load file list from the NAR file

• Inform the main thread via a channel that the NAR is now available, so that it can start answering FUSE requests on it

• Answer FUSE requests that were delegated to the task during the time it took to load the NAR

No data structure is shared using locking such as mutexes between the main thread and the loader thread, as this would risk a performance penalty on unrelated FUSE requests. We also avoid using shared lockless datastructures. Only inode number allocation is shared between all components using an atomic counter, to ensure that they are globally unique.

Support for writes

This section describes planned work which is not yet implemented.

Magic Nix VFS has support for the Nix daemon writing to the Nix store, for example when it builds a new path or when it copies a path from a cache server or another machine.

When Nix writes to /nix/store, all writes are buffered in RAM on the main thread, acting like a tmpfs.

Magic Nix VFS detects when the Nix daemon has finished writing to a path, as this is when all files and directories in that path are changed to read-only. At that point, the main thread will send a copy of all the data to a worker thread that will write it out to a NAR file on disk. When the worker thread has finished writing out the NAR file, it will inform the main thread that will then remove all actual content in RAM and replace it by offsets in the file on disk.

Integration with the NixOS boot process

To run a full NixOS system with Magic Nix VFS, tight integration with the boot process is required. In particular, we need to modify the initrd scripts so that Magic Nix VFS is mounted on /nix/store during the first boot phase.

The boot process with Magic Nix VFS looks as follows:

1. The bootloader loads a kernel and initrd.

2. The kernel starts up and launches a stage1 init script from the initrd.

3. The stage1 init script configures network connectivity.

4. The stage1 init script mounts local disks if any. In a pure netboot scenario without local disks, a tmpfs is mounted on / instead.

5. The stage1 init script mounts Magic Nix VFS on /nix/store

6. The stage1 init script runs the stage2 init script specified in the init= kernel command line parameter. This script is a path under /nix/store, so it will be loaded from Magic Nix VFS.

The actual modifications to the stage1 init that make this process possible are visible in module/magicnixvfs.nix in our Git repo.

Note that currently this process only works with the old-style initrd composed of shell scripts, and not with the new-style initrd based on systemd.

Integration with the Nix daemon

TODO

NixOS module options

See this section for high-level documentation on how to import and use the magicnixvfs module in your system.

boot.magicnixvfs.enable

Whether to enable Enable booting using Magic Nix VFS.

Type: boolean

Default:

false

Example:

true

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix

boot.magicnixvfs.package

Magic Nix VFS package to use for booting.

Type: package

Default:

<derivation magicnixvfs-0.1.0>

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix

boot.magicnixvfs.caches

URIs of Nix binary caches

Type: list of string

Default:

[
  "https://cache.nixos.org/"
]

Example:

''
  [ "https://cache.nixos.org" ]
''

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix

boot.magicnixvfs.dnsPreResolve

Whether to resolve hostnames of cache servers when mounting, instead of when they are used.

Use this if you are booting from the network. Nameservers must be accessible in the initrd when magicnixvfs is mounted.

Type: boolean

Default:

false

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix

boot.magicnixvfs.statePath

Magic Nix local state directory, can be a tmpfs or a disk partition.

Make sure that the mountpoint that contains this path has the neededForBoot = true configuration option.

Type: string

Default:

"/var/lib/magicnixvfs"

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix

boot.magicnixvfs.trustedPublicKeys

Trusted public keys of Nix binary caches

Type: list of string

Default:

[
  "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
]

Example:

''
  [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ]
''

Declared by:

• /nix/store/0c7qm6wkgdg9dd3ws34vvg9xacdwa0dq-source/flake.nix