Home |

Prebuilding and Running AWS Instances with NixOS, GitHub Actions, and Terraform


The typical strategy for automatically launching Linux VMs in AWS looks something like this:

  1. Use Packer to create an EC2 instance, connect to it via SSH, run commands to configure the machine, snapshot the EC2 as an AMI, and then terminate the instance.
  2. Use Terraform or another deployment script to launch a new EC2 instance using the AMI as a base.
  3. Configure user data (cloud-init) to run initialization scripts on first boot.

This process is brittle, requires multiple steps, including launching a new EC2 instance multiple times. Instead, I like to prebuild a NixOS AMI and let Terraform import the image into AWS and launch the machine in a single action.

Prebuilding the Image

In my flake, I use nixos-generators to create an x86-64 Amazon image from my pre-configured NixOS configuration. I use nix build to create a .vhd disk image file which can be uploaded to S3 afterwards or picked up by Terraform and pushed to S3.

Importing the AMI

In order to generate a valid AMI, the .vhd file must be imported directly from S3. With Terraform, this can be done using the aws_ebs_snapshot_import resource.

Once the AMI is imported, you can reference it in a basic aws_instance resource just like any other EC2 launch.

Putting It All Together

Using a GitHub Actions workflow you can perform the following steps to automatically launch everything in one swoop:

  1. Enable KVM in GitHub Actions in order to generate the image locally.
  2. Build the image with nix build, using a cache if available.
  3. Upload the image to S3, which can also be done in Terraform instead. One reason to include this step outside of Terraform is, if you want to sometimes skip building the image to save time, Terraform doesn’t have to expect the image to always be there on disk.
  4. Run Terraform apply to import the image and launch the EC2 instance.
  5. Get the host IP and wait for SSH to be ready in order to push content that does not belong in the Nix store (i.e. secrets).
  6. Push secrets to the machine with SSH using the pre-generated SSH key placed in the config.
  7. The systemd services will automatically start up, including registering their domain names with Cloudflare.

The instance is now running on AWS and ready to send or receive traffic!