Search code examples
nixos

Nix store across multiple disks

Is there configuration to make nix use additional disks for its store? Lets say I have small physical disk where nix is installed and multiple unused network ones mounted /mnt/... I would like to use to not run out of space when doing nixos-rebuild switch. Ideal situation would be to store every other generation but the current one there so the system is bootable without them.

(I am not asking how to share disk between multiple nixos machines)

Solution

  • There are a few ways to do this, depending on the constraints (local/remote, reliable/intermittent, NixOS/non-NixOS, etc.). For example, I'm currently running Nix on a non-NixOS system without much disk space; so I'm setting up two Nix stores, one local for day-to-day commands, and another on a remote disk for large build jobs.

    Putting /nix/store on a different local disk

    If our only problem is that / is too small, but we have some other large drive available, then we can use that for our Nix store by simply mounting it at /nix/store. This is easier on non-NixOS systems, since they don't rely on /nix/store during boot, but is doable on NixOS too. There are various instructions for doing this, e.g.

    Combining local disks for more space

    Others have already pointed out that LVM, ZFS, RAID, etc. can be used to combine multiple disks. These can be good solutions, but have a few caveats:

    • The disks must be local devices, not remote
    • The disks must be reliable and available, e.g. if we combine a laptop's internal drive with a USB hard drive, we must keep that hard drive plugged in at all times.
    • We must format our disks specially for this purpose

    Combine arbitrary folders with a union filesystem

    We can avoid most of the constraints imposed by LVM/ZFS/etc. by using a union filesystem (at the cost of performance). There are many union filesystems, but my preferred one is mergerfs:

    • Instead of relying on disks, we combine multiple folders into a single mount point. That makes it independent of the underlying storage, e.g. we can combine local drives, remote Samba shares, rclone mounts of S3 buckets, etc.
    • We don't need to reformat any drives.
    • We can add and remove folders from the union on the fly, by just remounting with different options. For example, when inserting/removing a USB drive, or connecting/disconnecting our home LAN.
    • We can keep using the individual folders, even while they're part of a union mount.
    • Since the individual folders are not specially formatted, they can be used as-is by other systems (e.g. a remote folder can be used as-is by others systems that have no idea about our union mount shenanigans)
    • The choice of where to write is configurable, e.g. we can give a preference order (say, write to a USB drive when it's present), or write wherever has the most/least free space.
    • mergerfs has special "path preserving" options, which writes files with a common prefix to the same drive. This way, a store path (say, Python3) will be stored entirely in one of the unioned folders, rather than having parts scattered across multiple.
    • mergerfs-tools provides handy scripts for managing the underlying storage; e.g. balancing, replicating, etc. (it just runs rsync commands on the underlying folders!)

    Using multiple Nix stores via different store paths

    We can use the NIX_STORE_DIR env var to tell Nix commands to use a different path for their store. This is simple, and ensures we're only running a single Nix installation.

    However, this is usually a bad idea, since that different path will be observable to all of the derivations; hence we can't use anything that's been built against the normal /nix/store location, which includes everything in pretty much all binary caches (the official Nix one, everything on Cachix, etc.)

    The following options are mostly documented on https://nixos.wiki/wiki/Nix_Installation_Guide#Installing_without_root_permissions

    Tricking Nix into seeing a different /nix/store

    Instead of asking Nix to use a different store path, we can instead run Nix in an environment where /nix/store points somewhere else. There are many ways to do this, but in all cases it's usually safest to use a whole different /nix (not just the store), and use a separate Nix installation (e.g. by running the usual Nix install script). Options to do this include:

    • Using chroot. In this case, you'll want a folder with the same entries as / (e.g. /etc, /tmp, etc.) and bind-mount all of them to their / equivalent; except (of course) for /nix, which will be your separate store. Note that bind mounts don't propagate anything that's mounted in a sub-folder, so you'll probably have to manually mount dev/pts and dev/shm in your chroot.

    • If you can't use chroot (or don't want to) then PRoot can be used instead. PRoot also integrates with qemu which should let us run executables for different CPU architectures (although I couldn't get that to work so far :( )

    • The nix-user-chroot tool will do basically the same as above; you just give it the path to your desired Nix store.

    • The nix-portable tool also does a similar thing, using PRoot, etc.

    • nix run --store can be used to make a different /nix available to sub-processes (e.g. we can use this to run bash, and hence whatever other commands we like)

    Use as a remote builder

    The above setup works fine for one-off builds, but requires manual use of Nix commands to copy closures back and forth to the "normal" system store. Instead, you could configure your chroot installation as a remote builder. That way, you can tell Nix to use it (either per command, or in your config file) and it will take care of copying derivations and build products back and forth. Also, this way you can garbage-collect your normal Nix store, without losing the copies that were built on the chroot store.

    This would require the chroot environment to be generally available, e.g. by logging in via SSH user, rather than just a one-off. This could be managed by a suitable SSH alias; or maybe a systemd service that sets up the chroot. At that point it may be easier to jump straight to a container/VM/etc. (since those have dedicated tooling)

    More heavyweight approaches to separate Nix installations

    If you don't mind bloat, but would like to leverage pre-existing tooling, you could upgrade the above chroot approach to a full-blown container (using systemd, nerdctl, podman, etc.) or even a virtual machine (qemu/kvm, VirtualBox, etc.) or emulator (if you want to build for a different architecture, and PRoot doesn't work for you).

    This would also let you use pre-built images, rather than running the installer yourself. One downside is the added complexity of integrating it with the host; e.g. it may require separate user accounts, full-blown networking, "secret" keys, etc.