Internals

Magic Nix VFS implements a FUSE-based filesystem that is mounted under /nix/store. It is divided in two parts:

• a single-threaded, synchronous FUSE server that runs only from metadata stored in RAM and tries to answer metadata requests on store files as fast as possible

• a multi-threaded I/O component based on a thread poool, that loads NAR archives from disk, downloads NAR archives as necessary from cache servers, and bundles up newly created paths into new NAR archives (that can also be uploaded to any configured remotes)

The single-threaded part keeps the following metadata in RAM:

• for recently accessed and currently referenced NAR archives, the list of all files in the archive (and their associated inode number) as well as the position of their actual content in the archive

• information about paths that are currently being loaded from disk or downloaded from the network

• for paths that are being written to by Nix during a build phase, the entire content of the path similarly to a tmpfs. Once the build finishes and the path becomes fully read-only, it is bundled in a new NAR archive and the actual file contents are removed from RAM (becoming like the first case).

All of this information is NOT guarded by a mutex lock, it is fully local to this thread to enable answering FUSE requests as fast as possible.

When reading data from a Nix path which is served from a NAR archive, several code paths are possible:

• for metadata requests such as stat or readdir on paths that already have their metadata loaded in RAM, the single threaded part can answer immediately.

• for metadata requests on paths that are not yet loaded, a task is pushed to the thread pool to retrieve the path and load its metadata in RAM. All requests for this path are transferred to the offloaded task while the load is in progress, so that the single-threaded component can continue processing requests for other paths. The offloaded task will answer all queued requests once the path is loaded.

• for the read system call which is used to read actual data from files within the NAR archives, the single-threaded part looks up the corresponding offsets to read from in the metadata in RAM, and then launches a new task on the thread pool to read that data and answer the request later. The syngle-threaded part keeps on answering other requests in the meantime.

When the task finishes loading the metadata for a store path, it informs the single-threaded part via a channel, which is checked before answering any FUSE request. No data structure is shared between the two parts using locking such as mutexes, as this would risk a performance penalty on unrelated FUSE requests. We also avoid using shared lockless datastructures. Only inode number allocation is shared between all components using an atomic counter. As an optimization measure, inode numbers are allocated in contiguous blocks of 256.

The thread pool will actually be divided in two pools to separate jobs that might access the network and jobs that only access the local disk, to guarantee that local disk access is always possible and is not blocked by networking.