Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Overview

Magic Nix VFS is a virtual filesystem implemented using FUSE that provides /nix/store instead of an actual disk partitions. It is backed by NAR archives on an actual physical disk, and makes the files packaged in these archives visible to users as regular files which can be used directly. Magic Nix VFS can automatically download and serve NAR archives for any paths found in configured cache servers. For instance, if you enable the cache.nixos.org cache server, then all programs built by the NixOS project will be directly visible in /nix/store and can be launched immediately, without an invocation of the Nix toolchain to download the closure.

Magic Nix VFS proposes a new trade-off where we pay for some RAM usage and upfront network traffic, and in return we optimize certain scenarios and unlock new use cases for the Nix/NixOS ecosystem.

Scenario 1: an HPC cluster is configured using NixOS. A head node contains the Nix store, and all compute nodes want to access paths from this store directly in /nix/store. With current tools, a common solution would be to mount /nix/store from the head node using NFS. This incurs a significant penalty especially when reading many small files, for instance when doing an evaluation of Nixpkgs (several thousands of .nix files). With Magic Nix VFS, the head node can be configured as a cache on the compute nodes. Instead of requesting files one by one to the NFS server, incurring a round-trip-time latency each time, each time a store path is accessed, the entier path would be prefetched from the head node, and subsequent access to files inside the path would be local and thus very fast. To enable the full Nix toolchain to run from compute enodes, all builds can be delegated to the head node using the experimental mounted-ssh-store feature of Nix, or simply by proxying the Nix daemon socket to the head node via a TCP connection.

Scenario 2: a distributed cluster of servers is configured to run NixOS. All nodes might want to build Nix paths, as part of their reconfiguration and upgrade process. We want to share paths that are built automatically between the nodes. Moreover, an orchestrator is used that automatically places jobs running from the Nix store on the various nodes. We want the files for the various jobs to be immediately visible in /nix/store when a job needs to be launched. Magic Nix VFS can be used to serve paths in /nix/store from cache.nixos.org plus a local cache (such as a Garage cluster). Each time a Nix derivation is built on any node, Magic Nix VFS will upload it to the local cache so that it is shared between nodes. Without Magic Nix VFS, it would be much more cumbersome to share builds between nodes. Moreover invoking the full Nix toolchain and doing a full evaluation would be required when launching a job (to ensure that the whole closure is present in the local nix store). Instead, Magic Nix VFS lazily fetches paths that are already fully built as they are needed.

Scenario 3: a public cache cache.magicnix.net is set up that contains pre-build NixOS systems. A minimal dd image with just a kernel and an initrd is provided, that allows booting into these systems by mounting /nix/store as a magic Nix VFS that uses the cache to lazily download necessary files to boot and run the system as it is used. The dd image itself would likely be less than 50MB, so burning it to a USB pendrive would be very fast.

Usage

Setting up a NixOS system that runs fully on Magic Nix VFS is still a bit tricky, so for now we only provide a demo system as an ISO image.

Magic Nix VFS demo

Requirements:

  • a computer or VM
  • with a wired Internet connection with automatic DHCP configuration
  • and at least 8GB of RAM (for the destkop variant) or 2GB (for the CLI variant)

Download:

Instructions:

  1. Download the ISO file above.

  2. If booting a physical computer, burn ISO to a CD-ROM or USB pendrive.

  3. Boot computer or VM from this disk.

  4. Select the variant of the demo to run: full desktop with Cinnamon, or CLI-only. The CLI variant will boot faster but has less functionality.

  5. If using the CLI variant, login with username root and password root.

  6. Explore the system.

Running programs:

If you are running the desktop version of the demo, note that no extra programs are installed by default. However, all binary packages ever built by the NixOS project and present on cache.nixos.org can be run simply by launching their full path. For instance, try launching one of the following commands:

  • /nix/store/kq5pl1xnpzmkh1x6dzy857l0929bkgx5-thunderbird-152.0.1/bin/thunderbird
  • /nix/store/4b7ydwmf41lzcha054wf4m5nki4fb2ml-inkscape-1.4.4/bin/inkscape

Internals

Magic Nix VFS implements a FUSE-based filesystem that is mounted under /nix/store. It is divided in two parts:

  • a single-threaded, synchronous FUSE server that runs only from metadata stored in RAM and tries to answer metadata requests on store files as fast as possible

  • a multi-threaded I/O component based on a thread poool, that loads NAR archives from disk, downloads NAR archives as necessary from cache servers, and bundles up newly created paths into new NAR archives (that can also be uploaded to any configured remotes)

The single-threaded part keeps the following metadata in RAM:

  • for recently accessed and currently referenced NAR archives, the list of all files in the archive (and their associated inode number) as well as the position of their actual content in the archive

  • information about paths that are currently being loaded from disk or downloaded from the network

  • for paths that are being written to by Nix during a build phase, the entire content of the path similarly to a tmpfs. Once the build finishes and the path becomes fully read-only, it is bundled in a new NAR archive and the actual file contents are removed from RAM (becoming like the first case).

All of this information is NOT guarded by a mutex lock, it is fully local to this thread to enable answering FUSE requests as fast as possible.

When reading data from a Nix path which is served from a NAR archive, several code paths are possible:

  • for metadata requests such as stat or readdir on paths that already have their metadata loaded in RAM, the single threaded part can answer immediately.

  • for metadata requests on paths that are not yet loaded, a task is pushed to the thread pool to retrieve the path and load its metadata in RAM. All requests for this path are transferred to the offloaded task while the load is in progress, so that the single-threaded component can continue processing requests for other paths. The offloaded task will answer all queued requests once the path is loaded.

  • for the read system call which is used to read actual data from files within the NAR archives, the single-threaded part looks up the corresponding offsets to read from in the metadata in RAM, and then launches a new task on the thread pool to read that data and answer the request later. The syngle-threaded part keeps on answering other requests in the meantime.

When the task finishes loading the metadata for a store path, it informs the single-threaded part via a channel, which is checked before answering any FUSE request. No data structure is shared between the two parts using locking such as mutexes, as this would risk a performance penalty on unrelated FUSE requests. We also avoid using shared lockless datastructures. Only inode number allocation is shared between all components using an atomic counter. As an optimization measure, inode numbers are allocated in contiguous blocks of 256.

The thread pool will actually be divided in two pools to separate jobs that might access the network and jobs that only access the local disk, to guarantee that local disk access is always possible and is not blocked by networking.