Bringing up a NixOS VM in 10 minutes using nixos-anywhere

Getting started with nix is something of a challenge; there are multiple reasons for this — the distinction between nix the language, NixOS the Operating System and nix the package manager is not sufficiently clear (although a good overview of the main concepts is here), documentation is always behind the current state and there are many folks moving the nix ecosystem forward in different directions making it difficult to see the bigger picture.

Even the basic process of bringing up a basic VM running NixOS to kick the tyres is not so straightforward: the installation guide has a strong emphasis on creating boot images which feels very old school. Ultimately, it’s more hassle than it’s worth for some folks. (One point which is worth noting here is that the widely used cloud-init solution for initializing other distributions is quite at odds with the NixOS model which has a stronger focus on being able to declare full system state — this is one reason why it’s not so straightforward to boot NixOS on cloud providers).

Here I describe an approach to bring up a NixOS VM in 10 minutes to get some feel for what it’s like — all that is required is a running docker engine (probably on your local system) and an SSH-accessible test VM running an arbitrary linux distribution which will become your NixOS VM; we’ll be using the amazing nixos-anywhere tool which can be used to hijack an existing VM and deploy NixOS on it.

I’ve put together this git repo to help. This approach is much simpler than my previous approach.

Prerequisites

You will need:

• a functioning docker engine
• docker-compose
• an existing VM running a modern linux distro — I used a minimal ubuntu image. You will need to be able to log in to this machine via SSH to an account on which has root privileges. The nixos-anywhere tool indicates that 1.5GB of spare memory is required during the switchover and it’s probably sensible to have at least a 30GB disk.

nixos-anywhere

nixos-anywhere is a tool which was developed primarily by the numtide folks and can be used to deploy NixOS to existing remote machines; it also supports subsequent management of these machines via nix/git but that is not the focus here.

The basic operation is to log into the remote machine, use kexec to run a new kernel instead of the existing kernel and with this new kernel, reformat the disk and install a new NixOS distribution with the given configuration. It can deploy a fully functional NixOS distribution in a few minutes. Obviously, it needs to be used with caution and in particular, it should not be used on systems which are running existing workloads.

Here, we are focused on the NixOS 101 scenario where someone just wants to bring up a VM and make some changes to it to experiment with and understand a bit about the NixOS model. The intent is that the VM is essentially a disposable VM which does not run any current workload and will almost certainly be deleted subsequently. It is worth noting that nixos-anywhere was not specifically designed for this case where the most natural design would be to support system deployment and then the deployment tool could be deleted; the nixos-anywhere model is such that the deployment and management system remains and is required for remote management of the machine. That said, it’s very easy to get your first NixOS VM up and running with it.

Bringing up the deployment container

Clone the git repo on your host machine and run the following:

$ cd docker
$ docker-compose run --rm nixos-anywhere

This will create a new docker volume for this work, which is mapped to the /root directory within the container — this enables the content stored there to be persisted if the container is no longer running.

You will get a shell inside the container which has the nix toolchain installed. For this work, nix flake support is required so you will need to do the following:

bash-5.2# mkdir -p /root/.config/nix
bash-5.2# echo experimental-features = nix-command flakes > /root/.config/nix/nix.conf
bash-5.2# # the following is required on an aarch64 machine building x86_64 binaries
bash-5.2# echo filter-syscalls = false > /root/.config/nix/nix.conf

To test that this is working properly, create a nix shell which contains jq as follows:

bash-5.2# jq
bash: jq: command not found
bash-5.2# nix shell nixpkgs#jq
bash-5.2# jq
jq - commandline JSON processor [version 1.7.1]

Usage: jq [options] <jq filter> [file...]
 jq [options] --args <jq filter> [strings...]
 jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified except for formatting. For more advanced filters see
the jq(1) manpage ("man jq") and/or https://jqlang.github.io/jq/.

Example:

 $ echo '{"foo": 0}' | jq .
 {
  "foo": 0
 }

For listing the command options, use jq --help.
bash-5.2# 

Note that in the above there are two layers of shells — the outer shell does not have jq ; the inner shell which is instantiated with the nix shell command downloads jq from the nix cache and makes it available in the shell. (Similarly it’s possible to create a shell containing jq ,curl and neovim using nix shell nixpkgs#jq nixpkgs#curl nixpkgs#neovim). Currently, nix does not make it straightforward to differentiate between these different shell levels.

SSH keys

For this to work, it’s necessary to have an SSH key on the remote VM such that it can be accessed from the container; the simplest way to do this is to create a (disposable) SSH key in the container which is put on the remote VM.

bash-5.2# ssh-keygen
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:fMtOZkh9v2VJw82deJ33TjlIz3K7lOa592nqM8o7GOg root@13af6041781f
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|                 |
|                 |
|       . .    o.*|
|        S o .o B*|
|       o = o..* B|
|      . . O  o.@=|
|       E *.. o*B*|
|          .+=o*BB|
+----[SHA256]-----+
bash-5.2# cat /root/.ssh/id_ed25519.pub 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMTN8Tz+K+wQU3sox3hp+j0jP0LSklZa6tcF2Hk4u0QS root@13af6041781f
bash-5.2# 

Copy the public key generated and put it in /root/.ssh/authorized_keys on the VM. You can do this by running cat /root/.ssh/id_ed25519.pubto print the public key and copy and pasting it into the VM. Test that it works by running ssh root@<VM-name-or-ip-address> from inside the container.

Running nixos-anywhere

With a container running nix which can access the remote VM, we are now ready to deploy NixOS to it.

First, clone the repo inside the container.

bash-5.2# cd /root
bash-5.2# git clone https://github.com/seanrmurphy/nixos-in-10-minutes.git

Next, log in to the remote VM and determine the name of the storage device — this can be done using lsblk. This is required as an input to the disk formatting operation which is performed by nixos-anywhere.

root@ubuntu:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0      11:0    1  364K  0 rom  
vda     254:0    0 27.9G  0 disk 
├─vda1  254:1    0 27.8G  0 part /
├─vda14 254:14   0    4M  0 part 
└─vda15 254:15   0  106M  0 part /boot/efi
root@ubuntu:~# 

In the above, the device is called vda; it is often sda depending on where your VM lives — just make sure to have consistency between the name of the block device as seen from the VM and as specified in the nixos-anywhere/disk-config.nix file (on the line device = lib.mkDefault “/dev/sda”).Note that you may need to run a shell with an editor when making changes inside the container — usenix shell nixpkgs#vim to do this.

Have a look at the file nixos-anywhere/configuration.nix — this describes the NixOS configuration which will be applied to the remote machine. You should modify the SSH key used for logging into user accounts from this file; otherwise you’ll likely end up with a working NixOS deployment but you’re unable to access it. It’s a good idea to add a couple of pubkeys to the root account here — one which is one of your standard keys and one which is the key which has been generated in the container above.

When you are ready, run nixos-anywhere to deploy NixOS to the remote machine. This will take a couple of minutes and automate the entire process of setting up NixOS on the remote machine with the configuration specified in nixos-anywhere/configuration.nix. You can do this as follows:

bash-5.2# cd nixos-anywhere
bash-5.2# nix run github:nix-community/nixos-anywhere -- --flake .#nixos-anywhere-vm root@<vm-ip-address> 

Once it has finished, you can log into the remote machine and check the running Operating System — the upgrade has changed the host keys on the VM so you will likely need to modify the known_hosts file to remove the old host keys.

bash-5.2# ssh root@192.168.122.50
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:O6L3LmrK0kNtER5BjNQS4Y4RyHfc6kN3wN/2MDXjImE.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:4
Host key for 192.168.122.50 has changed and you have requested strict checking.
Host key verification failed.
bash-5.2# nvim ~/.ssh/known_hosts
bash-5.2# ssh root@192.168.122.50
The authenticity of host '192.168.122.50 (192.168.122.50)' can't be established.
ED25519 key fingerprint is SHA256:O6L3LmrK0kNtER5BjNQS4Y4RyHfc6kN3wN/2MDXjImE.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.50' (ED25519) to the list of known hosts.

[root@nixos:~]# ls -l
total 0

[root@nixos:~]# cat /etc/os-release 
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="23.11.20230921.e124831"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 23.11 (Tapir)"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="23.11 (Tapir)"
VERSION_CODENAME=tapir
VERSION_ID="23.11"

[root@nixos:~]#

To get a feel for how to run NixOS, uncomment some of the settings in the configuration.nix file — add an extra service, add an extra user etc. To make the changes take effect, simply run the same nixos-anywhere command as above from the docker container and the system state will be modified.

If you want to go further, check out what packages can be installed here and, for example, check the different options supported here.

Final comments

nixos-anywhere is a great tool for supporting remote systems management; it’s a good current solution for getting up and running with NixOS quickly to give it a spin.

You may need to remove the docker volume once you’re finished working with this.