burrow/docs/NIX_CACHE.md

Burrow Nix Cache

nix.burrow.net is the Burrow Nix binary cache surface for forge jobs and workers.

The host layout is:

• Garage stores objects on the forge host's Garage data volume.
• Attic serves the Nix binary cache API at nix.burrow.net.
• Attic stores NAR chunks in the Garage attic bucket through the local S3 API.
• Workers consume https://nix.burrow.net/burrow as a substituter once the cache public key is configured.

Garage is not a multi-backend object sync manager. It can replicate across Garage nodes, but it does not write the same object to GCS and another provider as independent storage backends. Multi-cloud backup should be a separate mirror or export job from Garage buckets to GCS and later to additional providers.

Host Bootstrap

services.burrow.garage creates a host-local environment file at /var/lib/burrow/garage/env. The file is not stored in Git or the Nix store. It contains:

• Garage RPC secret
• Garage access keys for Attic, release artifacts, package repositories, and read-only backups
• Attic JWT signing secret
• AWS-compatible variables used by Attic to talk to Garage

burrow-garage-bootstrap.service creates the initial single-node Garage layout, imports the generated keys, and creates the attic, burrow-releases, and burrow-packages buckets. It grants the backup key read-only access to those three buckets so CI can mirror them without receiving owner/write access to Garage.

services.burrow.nixCache enables Attic, points it at Garage, exposes it through Caddy, creates a public burrow cache, and writes two host-local tokens:

• /var/lib/burrow/nix-cache/admin-token: bootstrap/admin recovery token
• /var/lib/burrow/nix-cache/ci-push-token: scoped pull/push token for the burrow cache

Worker Configuration

Scripts/ci/ensure-nix.sh uses the Burrow cache only when BURROW_NIX_CACHE_PUBLIC_KEY is present:

BURROW_NIX_CACHE_URL=https://nix.burrow.net/burrow
BURROW_NIX_CACHE_PUBLIC_KEY=<cache public key from attic cache info>

Without the public key, CI falls back to cache.nixos.org only. This avoids configuring an unsigned or untrusted substituter.

After deployment, get the public key from the host:

attic cache info local:burrow

Then set BURROW_NIX_CACHE_PUBLIC_KEY in Forgejo variables.

Seal the scoped push token from /var/lib/burrow/nix-cache/ci-push-token into Forgejo/OpenBao as BURROW_NIX_CACHE_PUSH_TOKEN. The Cache: Publish Nix workflow runs Scripts/ci/publish-nix-cache.sh, builds the selected flake outputs, and pushes their closures to Attic. It skips cleanly when the push token is absent unless BURROW_NIX_CACHE_REQUIRED=true.

Backups

The Garage attic bucket is backed up to the private GCS bucket burrow-net-nix-cache. Release artifacts and package repositories are backed up to their public GCS buckets. The scheduled Backup: Garage Storage workflow runs Scripts/ci/backup-garage-to-gcs.sh with:

BURROW_GARAGE_ENDPOINT=https://objects.burrow.net
AWS_ACCESS_KEY_ID=<GARAGE_BACKUP_ACCESS_KEY_ID from the host env>
AWS_SECRET_ACCESS_KEY=<GARAGE_BACKUP_SECRET_ACCESS_KEY from the host env>
BURROW_NIX_CACHE_GCS_BUCKET=burrow-net-nix-cache

The workflow authenticates to Google through Authentik-backed WIF and uses gcloud storage rsync. It does not use rclone or Google service-account JSON keys. Destination deletes are disabled by default; set BURROW_GARAGE_BACKUP_DELETE=true only for an intentional prune run.