burrow/nixos

Burrow Forge Runbook

This directory contains the Burrow forge host definition and the Hetzner bootstrap shape for burrow-forge.

Mail hosting is intentionally not part of this NixOS host in the current plan. Burrow's first mail path is Forward Email with Burrow-owned custom S3 backups; see docs/FORWARDEMAIL.md.

Files

• hosts/burrow-forge/default.nix: host entrypoint
• modules/burrow-forge.nix: Forgejo, Caddy, PostgreSQL, and admin bootstrap module
• modules/burrow-forge-runner.nix: Forgejo Actions runner and agent identity bootstrap
• upstream compatible.systems/conrad/nsc-autoscaler: Namespace-backed ephemeral Forgejo runner module consumed via the Burrow flake input
• modules/burrow-authentik.nix: minimal Authentik IdP for Burrow control planes
• modules/burrow-headscale.nix: Headscale control plane rooted in Authentik OIDC
• ../secrets.nix: agenix recipient map for tracked Burrow forge secrets
• hetzner-cloud-config.yaml: desired Hetzner host shape
• keys/contact_at_burrow_net.pub: initial operator SSH public key
• keys/agent_at_burrow_net.pub: automation SSH public key
• ../Scripts/hetzner-forge.sh: Hetzner inventory and replace workflow
• ../Scripts/nsc-build-and-upload-image.sh: temporary Namespace builder -> raw image -> Hetzner snapshot
• ../Scripts/bootstrap-forge-intake.sh: copy the Forgejo bootstrap password and agent SSH key into /var/lib/burrow/intake/
• ../Scripts/check-forge-host.sh: verify Forgejo, Caddy, the local runner, optional NSC services, and optional Tailnet services after boot
• ../Scripts/cloudflare-upsert-a-record.sh: upsert DNS-only Cloudflare A records for Burrow host cutovers
• ../Scripts/forge-deploy.sh: remote nixos-rebuild entrypoint for the forge host
• ../Scripts/provision-forgejo-nsc.sh: render Burrow Namespace dispatcher/autoscaler runtime inputs and ensure the default Forgejo scope exists
• ../Scripts/seal-forgejo-nsc-secrets.sh: encrypt forgejo-nsc runtime inputs into the agenix secrets consumed by burrow-forge

Intended Flow

1. Build and upload the raw NixOS image with Scripts/hetzner-forge.sh build-image or Scripts/nsc-build-and-upload-image.sh.
2. Recreate burrow-forge from the latest labeled snapshot with Scripts/hetzner-forge.sh recreate-from-image --yes.
3. Run Scripts/bootstrap-forge-intake.sh to place the Forgejo bootstrap password file and automation SSH key under /var/lib/burrow/intake/.
4. Let burrow-forgejo-bootstrap.service create or rotate the initial Forgejo admin account.
5. Let burrow-forgejo-runner-bootstrap.service register the self-hosted Forgejo runner and seed Git identity as agent <agent@burrow.net>.
6. Run Scripts/provision-forgejo-nsc.sh locally to refresh intake/forgejo_nsc_token.txt, intake/forgejo_nsc_dispatcher.yaml, and intake/forgejo_nsc_autoscaler.yaml.
7. Run Scripts/seal-forgejo-nsc-secrets.sh to encrypt those runtime inputs into the agenix secrets used by burrow-forge.
8. Ensure /var/lib/agenix/agenix.key exists on the host, encrypt secrets/infra/authentik.env.age, secrets/infra/authentik-google-client-id.age, secrets/infra/authentik-google-client-secret.age, secrets/infra/forgejo-oidc-client-secret.age, secrets/infra/headscale-oidc-client-secret.age, secrets/infra/forgejo-nsc-token.age, secrets/infra/forgejo-nsc-dispatcher-config.age, and secrets/infra/forgejo-nsc-autoscaler-config.age, and let agenix materialize them under /run/agenix/.
9. Use Scripts/cloudflare-upsert-a-record.sh to point git.burrow.net, burrow.net, auth.burrow.net, ts.burrow.net, and nsc-autoscaler.burrow.net at the host with Cloudflare proxying disabled for ACME.
10. Use Scripts/forge-deploy.sh --allow-dirty for subsequent remote nixos-rebuild runs from the live workspace.
11. Configure Forward Email custom S3 backups for burrow.net and burrow.rs out-of-band with Tools/forwardemail-custom-s3.sh.

Current Constraints

• burrow-forge is live on NixOS in hel1 at 89.167.47.21.
• services.forgejo-nsc now expects agenix-backed runtime inputs at /run/agenix/burrowForgejoNscToken, /run/agenix/burrowForgejoNscDispatcherConfig, and /run/agenix/burrowForgejoNscAutoscalerConfig.
• Authentik and Headscale secrets now live in tracked agenix blobs under secrets/infra/ and decrypt to /run/agenix/ on the forge host.
• Public Burrow forge cutover completed on March 15, 2026:
  • burrow.net, git.burrow.net, and nsc-autoscaler.burrow.net now publish public A records to 89.167.47.21
  • HTTP redirects to HTTPS on all three names
  • https://burrow.net returns the root forge landing response
  • https://git.burrow.net returns the live Forgejo front door
  • https://nsc-autoscaler.burrow.net terminates TLS on Caddy and returns the expected application-level 404 for /
• The Cloudflare token currently in intake/cloudflare-token.txt is an account-scoped token: POST /accounts/<account>/tokens/verify succeeds, while POST /user/tokens/verify returns Invalid API Token.
• burrow.rs still resolves publicly to a Vercel DEPLOYMENT_NOT_FOUND response.
• Both domains publish Forward Email MX/TXT records.
• Forward Email custom S3 is live on both domains against the Hetzner burrow bucket and the public regional endpoint https://hel1.your-objectstorage.com.
• The current Hetzner account contains both:
  • the older Ubuntu bootstrap server in hil
  • the live burrow-forge NixOS server in hel1
• The remaining forge work is follow-on product/integration work, not host bring-up, mail backup wiring, or public DNS cutover.