002bd382e9
Some checks failed
Cache: Publish Nix / Publish Nix Cache (push) Waiting to run
Build Rust / Cargo Test (push) Successful in 4m14s
Build Site / Next.js Build (push) Failing after 6s
Infra: OpenTofu / OpenTofu (grafana) (push) Successful in 5s
Lint Governance / BEP Metadata (push) Successful in 0s
Build: Android / Android Rust Core Stub (push) Failing after 23s
88 lines
3.4 KiB
Markdown
88 lines
3.4 KiB
Markdown
# Burrow Nix Cache
|
|
|
|
`nix.burrow.net` is the Burrow Nix binary cache surface for forge jobs and
|
|
workers.
|
|
|
|
The host layout is:
|
|
|
|
- Garage stores objects on the forge host's Garage data volume.
|
|
- Attic serves the Nix binary cache API at `nix.burrow.net`.
|
|
- Attic stores NAR chunks in the Garage `attic` bucket through the local S3 API.
|
|
- Workers consume `https://nix.burrow.net/burrow` as a substituter once the
|
|
cache public key is configured.
|
|
|
|
Garage is not a multi-backend object sync manager. It can replicate across
|
|
Garage nodes, but it does not write the same object to GCS and another provider
|
|
as independent storage backends. Multi-cloud backup should be a separate mirror
|
|
or export job from Garage buckets to GCS and later to additional providers.
|
|
|
|
## Host Bootstrap
|
|
|
|
`services.burrow.garage` creates a host-local environment file at
|
|
`/var/lib/burrow/garage/env`. The file is not stored in Git or the Nix store.
|
|
It contains:
|
|
|
|
- Garage RPC secret
|
|
- Garage access keys for Attic, release artifacts, package repositories, and
|
|
read-only backups
|
|
- Attic JWT signing secret
|
|
- AWS-compatible variables used by Attic to talk to Garage
|
|
|
|
`burrow-garage-bootstrap.service` creates the initial single-node Garage layout,
|
|
imports the generated keys, and creates the `attic`, `burrow-releases`, and
|
|
`burrow-packages` buckets. It grants the backup key read-only access to those
|
|
three buckets so CI can mirror them without receiving owner/write access to
|
|
Garage.
|
|
|
|
`services.burrow.nixCache` enables Attic, points it at Garage, exposes it through
|
|
Caddy, creates a public `burrow` cache, and writes two host-local tokens:
|
|
|
|
- `/var/lib/burrow/nix-cache/admin-token`: bootstrap/admin recovery token
|
|
- `/var/lib/burrow/nix-cache/ci-push-token`: scoped pull/push token for the
|
|
`burrow` cache
|
|
|
|
## Worker Configuration
|
|
|
|
`Scripts/ci/ensure-nix.sh` uses the Burrow cache only when
|
|
`BURROW_NIX_CACHE_PUBLIC_KEY` is present:
|
|
|
|
```sh
|
|
BURROW_NIX_CACHE_URL=https://nix.burrow.net/burrow
|
|
BURROW_NIX_CACHE_PUBLIC_KEY=<cache public key from attic cache info>
|
|
```
|
|
|
|
Without the public key, CI falls back to `cache.nixos.org` only. This avoids
|
|
configuring an unsigned or untrusted substituter.
|
|
|
|
After deployment, get the public key from the host:
|
|
|
|
```sh
|
|
attic cache info local:burrow
|
|
```
|
|
|
|
Then set `BURROW_NIX_CACHE_PUBLIC_KEY` in Forgejo variables.
|
|
|
|
Seal the scoped push token from `/var/lib/burrow/nix-cache/ci-push-token` into
|
|
Forgejo/OpenBao as `BURROW_NIX_CACHE_PUSH_TOKEN`. The
|
|
`Cache: Publish Nix` workflow runs `Scripts/ci/publish-nix-cache.sh`, builds the
|
|
selected flake outputs, and pushes their closures to Attic. It skips cleanly
|
|
when the push token is absent unless `BURROW_NIX_CACHE_REQUIRED=true`.
|
|
|
|
## Backups
|
|
|
|
The Garage `attic` bucket is backed up to the private GCS bucket
|
|
`burrow-net-nix-cache`. Release artifacts and package repositories are backed up
|
|
to their public GCS buckets. The scheduled `Backup: Garage Storage` workflow
|
|
runs `Scripts/ci/backup-garage-to-gcs.sh` with:
|
|
|
|
```sh
|
|
BURROW_GARAGE_ENDPOINT=https://objects.burrow.net
|
|
AWS_ACCESS_KEY_ID=<GARAGE_BACKUP_ACCESS_KEY_ID from the host env>
|
|
AWS_SECRET_ACCESS_KEY=<GARAGE_BACKUP_SECRET_ACCESS_KEY from the host env>
|
|
BURROW_NIX_CACHE_GCS_BUCKET=burrow-net-nix-cache
|
|
```
|
|
|
|
The workflow authenticates to Google through Authentik-backed WIF and uses
|
|
`gcloud storage rsync`. It does not use rclone or Google service-account JSON
|
|
keys. Destination deletes are disabled by default; set
|
|
`BURROW_GARAGE_BACKUP_DELETE=true` only for an intentional prune run.
|