Overview

Magic Nix VFS is a virtual filesystem implemented using FUSE that provides /nix/store instead of an actual disk partitions. It is backed by NAR archives on an actual physical disk, and makes the files packaged in these archives visible to users as regular files which can be used directly. Magic Nix VFS can automatically download and serve NAR archives for any paths found in configured cache servers. For instance, if you enable the cache.nixos.org cache server, then all programs built by the NixOS project will be directly visible in /nix/store and can be launched immediately, without an invocation of the Nix toolchain to download the closure.

Magic Nix VFS proposes a new trade-off where we pay for some RAM usage and upfront network traffic, and in return we optimize certain scenarios and unlock new use cases for the Nix/NixOS ecosystem.

Scenario 1: an HPC cluster is configured using NixOS. A head node contains the Nix store, and all compute nodes want to access paths from this store directly in /nix/store. With current tools, a common solution would be to mount /nix/store from the head node using NFS. This incurs a significant penalty especially when reading many small files, for instance when doing an evaluation of Nixpkgs (several thousands of .nix files). With Magic Nix VFS, the head node can be configured as a cache on the compute nodes. Instead of requesting files one by one to the NFS server, incurring a round-trip-time latency each time, each time a store path is accessed, the entier path would be prefetched from the head node, and subsequent access to files inside the path would be local and thus very fast. To enable the full Nix toolchain to run from compute enodes, all builds can be delegated to the head node using the experimental mounted-ssh-store feature of Nix, or simply by proxying the Nix daemon socket to the head node via a TCP connection.

Scenario 2: a distributed cluster of servers is configured to run NixOS. All nodes might want to build Nix paths, as part of their reconfiguration and upgrade process. We want to share paths that are built automatically between the nodes. Moreover, an orchestrator is used that automatically places jobs running from the Nix store on the various nodes. We want the files for the various jobs to be immediately visible in /nix/store when a job needs to be launched. Magic Nix VFS can be used to serve paths in /nix/store from cache.nixos.org plus a local cache (such as a Garage cluster). Each time a Nix derivation is built on any node, Magic Nix VFS will upload it to the local cache so that it is shared between nodes. Without Magic Nix VFS, it would be much more cumbersome to share builds between nodes. Moreover invoking the full Nix toolchain and doing a full evaluation would be required when launching a job (to ensure that the whole closure is present in the local nix store). Instead, Magic Nix VFS lazily fetches paths that are already fully built as they are needed.

Scenario 3: a public cache cache.magicnix.net is set up that contains pre-build NixOS systems. A minimal dd image with just a kernel and an initrd is provided, that allows booting into these systems by mounting /nix/store as a magic Nix VFS that uses the cache to lazily download necessary files to boot and run the system as it is used. The dd image itself would likely be less than 50MB, so burning it to a USB pendrive would be very fast.