Internals

Inside Magic Nix VFS

High-level overview

Magic Nix VFS implements a FUSE-based filesystem that is mounted under /nix/store. It is divided in two main parts:

• A main thread that runs the FUSE server. This thread keeps metadata stored in RAM and tries to answer metadata requests on store files as fast as possible

• A multi-threaded I/O component based on a thread pool. The pool runs I/O jobs in the background, such as loading NAR archives from disk or downloading them from a cache server as necessary.

In practice, the thread pool is actually divided in two separate pools for performance reasons.

• A “small task” pool that answers read requests from NARs already on disk.

• A “big task” pool that loads new NARs, downloading them from the network if necessary.

Read path

The main thread keeps the following metadata in RAM:

• For recently accessed and currently referenced NAR archives, the list of all files in the archive (and their associated inode number) as well as the position of their actual content in the archive

• Information about paths that are currently being loaded by a background thread (from disk or downloaded from the network)

All of this information is NOT guarded by a mutex lock, it is fully local to this thread to enable answering FUSE requests as fast as possible.

When reading data from a Nix path which is served from a NAR archive, several code paths are possible:

• For metadata requests such as stat or readdir on paths that already have their metadata loaded in RAM, the single threaded part can answer immediately.

• For the read system call which is used to read actual data from files within the NAR archives, the main thread looks up in metadata the corresponding offset to read from, and then launches a new “small task” on the thread pool to do the actual read. The main thread keeps on answering other requests in the meantime.

  The task delegated to the thread pool consist simply of doing a read syscall on the NAR file at the given offset, and then replying to the FUSE request with the read data. The read request does not pass back through the main thread, the worker thread can answer it directly.

Dynamically loading new NARs

When the main thread recieves a metadata request on a path that is not yet loaded, a “big task” is pushed to the thread pool to retrieve the path and load its metadata in RAM. During the time when the store path is being loaded, all requests for this path are transferred to the offloaded task, so that the main thread can continue processing requests for other paths.

The offloaded tasks runs the following steps:

• Check whether the path is already present on disk

• If it isn’t, find the path on a cache server and download it

• Load the narinfo file, check it is consistent, and check the signature matches the content of the NAR file

• Load file list from the NAR file

• Inform the main thread via a channel that the NAR is now available, so that it can start answering FUSE requests on it

• Answer FUSE requests that were delegated to the task during the time it took to load the NAR

No data structure is shared using locking such as mutexes between the main thread and the loader thread, as this would risk a performance penalty on unrelated FUSE requests. We also avoid using shared lockless datastructures. Only inode number allocation is shared between all components using an atomic counter, to ensure that they are globally unique.

Support for writes

This section describes planned work which is not yet implemented.

Magic Nix VFS has support for the Nix daemon writing to the Nix store, for example when it builds a new path or when it copies a path from a cache server or another machine.

When Nix writes to /nix/store, all writes are buffered in RAM on the main thread, acting like a tmpfs.

Magic Nix VFS detects when the Nix daemon has finished writing to a path, as this is when all files and directories in that path are changed to read-only. At that point, the main thread will send a copy of all the data to a worker thread that will write it out to a NAR file on disk. When the worker thread has finished writing out the NAR file, it will inform the main thread that will then remove all actual content in RAM and replace it by offsets in the file on disk.

Integration with the NixOS boot process

To run a full NixOS system with Magic Nix VFS, tight integration with the boot process is required. In particular, we need to modify the initrd scripts so that Magic Nix VFS is mounted on /nix/store during the first boot phase.

The boot process with Magic Nix VFS looks as follows:

1. The bootloader loads a kernel and initrd.

2. The kernel starts up and launches a stage1 init script from the initrd.

3. The stage1 init script configures network connectivity.

4. The stage1 init script mounts local disks if any. In a pure netboot scenario without local disks, a tmpfs is mounted on / instead.

5. The stage1 init script mounts Magic Nix VFS on /nix/store

6. The stage1 init script runs the stage2 init script specified in the init= kernel command line parameter. This script is a path under /nix/store, so it will be loaded from Magic Nix VFS.

The actual modifications to the stage1 init that make this process possible are visible in module/magicnixvfs.nix in our Git repo.

Note that currently this process only works with the old-style initrd composed of shell scripts, and not with the new-style initrd based on systemd.

Integration with the Nix daemon

TODO