# `BEP-0009` - Release Infrastructure and Store Upload Pipeline

```text
Status: Draft
Proposal: BEP-0009
Authors: gpt-5.5
Coordinator: gpt-5.5
Reviewers: Pending
Constitution Sections: II, III, IV, V
Implementation PRs: Pending
Decision Date: Pending
```

## Summary

Burrow needs a release pipeline that behaves like the project infrastructure it is replacing: release eligibility is detected from forge build tags, build artifacts are produced per platform, upload lanes are separate from build lanes, and store credentials are explicit automation inputs rather than hidden local state.

The first implementation creates the release spine now: `release-if-needed` dispatches a build when HEAD has not been tagged as released, `build-release` packages current artifacts on Namespace-backed runners, and `publish-store-uploads` performs guarded upload steps for App Store Connect, Sparkle, Microsoft Store, and storage-backed public channels after Authentik-backed Google WIF authentication.

The same change also makes the forge itself the release authority: release tags prefer the Burrow Forgejo remote, dispatch helpers talk to Forgejo directly with a live-host fallback, runner identity is discoverable from persistent host state, repository mirror sync is disabled for the canonical Burrow repository, `nix.burrow.net` provides a Burrow-owned Nix cache for workers, and Grafana observability is bootstrapped with Nix-owned Prometheus/OpenTelemetry/Jaeger service state plus OpenTofu-owned dashboards.

## Motivation

- Burrow already controls its forge and runner infrastructure, but the release path only produced a Linux CLI tarball.
- Apple, Linux, Sparkle, and future Windows outputs need the same build-number and artifact layout so release evidence is inspectable.
- The Namespace account should be part of the live runner surface, including macOS and Windows lanes, instead of sitting idle behind unused configuration.
- Authentication and store credentials should be named and checked at lane boundaries so dedicated accounts can be added without rewriting workflows.

## Detailed Design

- Use lightweight `builds/<number>` tags as the authoritative release watermark.
- Keep `Scripts/version.sh status` as the release-needed check:
  - `clean` means HEAD is exactly the latest authoritative build tag.
  - `dirty` means the release workflow should create a new build number.
- Split workflows:
  - `.forgejo/workflows/release-if-needed.yml` checks release status and dispatches `build-release.yml`.
  - `.forgejo/workflows/build-release.yml` builds and stages artifacts.
  - `.forgejo/workflows/publish-store-uploads.yml` uploads staged artifacts to external stores and public channels.
- Stage artifacts under `dist/builds/<build-number>/<platform>/`.
- Publish generic release assets to a Forgejo release named `Build <number>`.
- Upload release artifacts through `Scripts/ci/upload-release-storage.sh`. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the `burrow-net-releases` GCS backup bucket unless `BURROW_RELEASE_GCS_BACKUP=false`.
- Publish signed package repository trees through `Scripts/ci/upload-package-storage.sh`. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the `burrow-net-packages` GCS backup bucket unless `BURROW_PACKAGE_GCS_BACKUP=false`.
- Publish selected Nix closures through `.forgejo/workflows/publish-nix-cache.yml` and `Scripts/ci/publish-nix-cache.sh`. The workflow logs into `nix.burrow.net` with the scoped `BURROW_NIX_CACHE_PUSH_TOKEN` generated by host bootstrap and skips when the token is absent unless explicitly required.
- Mirror Garage buckets through `.forgejo/workflows/backup-garage-storage.yml` and `Scripts/ci/backup-garage-to-gcs.sh`. The scheduled workflow uses the host-generated Garage read-only backup key, Authentik-backed Google WIF, and `gcloud storage rsync` to back up `burrow-releases`, `burrow-packages`, and `attic` to GCS.
- Build Apple artifacts on `namespace-profile-macos-large` through Bazel wrapper targets:
  - `//bazel/apple:release_ios_stamp`
  - `//bazel/apple:release_macos_stamp`
  These targets wrap the existing Xcode project and give the workflow one stable entrypoint that shares the same Bazel/Nix cache setup while Burrow decides whether deeper `rules_apple` targets are worth the migration.
- Use Namespace cache setup before Bazel/Nix work. Prefer the Namespace Bazel remote cache when `nsc` is authenticated and fall back to a local disk/repository cache when remote setup is unavailable.
  Cache setup is optional release acceleration: Bazel-only cache steps must not
  install filesystem-cache helpers such as `spacectl`, and Namespace cache
  probes must be timeout-bounded so Apple builds can continue without the
  remote cache. Apple workflows should call the fail-open cache helper directly
  instead of entering the full Nix development shell just to decide whether a
  cache is available. macOS runners should use the local Bazel disk/repository
  cache path and skip the `nsc` remote-cache auth probe, which is optional and
  can block without GNU timeout support.
- The Nix CI shell must not pin Apple builds to a Bazel major that disagrees
  with `.bazelversion`. Prefer Bazelisk or a Bazel 9 package so the release
  lane uses the repository-declared Bazel 9.1.0 toolchain.
- Moving tarball URLs are not acceptable for release-critical Nix inputs. The
  forge worker lockfile must resolve nixpkgs through a fixed Git revision so
  fresh workers do not fail when a branch tarball no longer matches a stored
  NAR hash.
- The CI shell should prefer currently supported Node runtimes when nixpkgs
  marks an older major insecure. Release workers must fail on product build
  errors, not on avoidable toolchain end-of-life policy checks.
- Bazel repository and disk caches must live outside the checked-out repository
  on macOS runners. Bazel 9 rejects a repository contents cache inside the main
  repo because it can create spurious build graph failures.
- Bazel Apple genrules must explicitly pass the Nix CI shell tool path and
  `PROTOC` into the action environment. Xcode build phases run with a reduced
  path, and the Rust Network Extension build depends on `protoc`.
- Forgejo release workflows must use the artifact action generation supported
  by the deployed forge. The v4 artifact actions assume GitHub's newer artifact
  service and fail on this Forgejo install before downstream signing/upload
  jobs can consume the staged Apple artifacts.
- The release secret decrypt action must be self-contained after the Nix
  bootstrap step. Linux Namespace signing jobs install Nix at runtime, so the
  bootstrap writes Nix profile directories to `GITHUB_PATH`, the decrypt action
  resolves Nix from those profile paths, and the action builds `.#age` before
  falling back to `.#agenix`. The decrypt step must not require Python before
  the Nix CI shell is active.
- Authentik-backed Google WIF must exchange tokens against Authentik's OAuth
  token endpoint, not a provider-slug child path. The helper should prefer an
  explicit `BURROW_AUTHENTIK_WIF_TOKEN_ENDPOINT`, then OIDC discovery, then the
  Authentik `/application/o/token/` fallback derived from the configured issuer.
- The Google WIF Authentik application is a machine-to-machine
  client-credentials provider. It must not default to a human group policy
  binding, because the token exchange has no interactive user and Authentik will
  reject the grant before Google WIF can validate the issued JWT.
- Forgejo should store the Authentik WIF credential as the client-credentials
  `client_secret` form value. For service-account authentication, that value is
  `base64("<service-account-username>:<app-password-token>")`; the helper must
  send it as form data, not HTTP Basic auth.
- Produce unsigned Apple validation artifacts when signing is absent, but require signing for an App Store requested iOS release and for a Sparkle tester release. Uploadable artifacts are produced only after provisioning profiles can be synced from App Store Connect credentials and the relevant Apple certificate is proven to match the selected Google KMS key.
- Resolve release credentials through age/agenix before signed Apple lanes run. Forgejo secrets should carry only the runner age identity fallback and the Authentik WIF client secret until OpenBao can mint the runner token directly. App Store Connect keys and signing certificates live as sealed files under `secrets/`.
- Persist the forge runner SSH key as `/var/lib/forgejo-runner-agent/age_keystore` and export `BURROW_RUNNER_AGE_IDENTITY_PATH` so jobs can resolve agenix identities without embedding long-lived material in every workflow.
- Add `Scripts/forgejo-dispatch.sh` and `Scripts/forgejo-dispatch-via-host.sh`:
  - local dispatch uses an explicit token, local intake token, or the sealed forgejo-nsc dispatcher config when available;
  - host fallback SSHes to the live forge and reads the agenix-materialized dispatcher config there;
  - neither path relies on a GitHub remote or GitHub workflow API.
- Default build tag reads and pushes to the Burrow Forgejo remote when that remote exists. `BURROW_VERSION_REMOTE` remains the override for unusual migrations.
- Keep the canonical Forgejo repo as an ordinary source repository, not a mirror. The host config disables new mirror creation and removes mirror/push-mirror rows for the canonical Burrow repository at activation time.
- Add `.forgejo/workflows/infra-opentofu.yml` and `Scripts/grafana-tofu.sh` for Grafana API-managed folders and dashboards. NixOS owns the Grafana service, SSO, reverse proxy, datasources, Prometheus, OpenTelemetry collector, Jaeger, secrets, and boot-time configuration; OpenTofu owns API-created dashboard objects.
- Sync provisioning profiles from App Store Connect in CI using the decrypted API key:
  - iOS App Store profiles for `net.burrow.app` and `net.burrow.app.network`.
  - macOS Developer ID profiles for the same bundle identifiers.
  Provisioning profile `.age` files are not part of the normal release path.
- Add a non-exportable Google KMS RSA-2048/SHA-256 key named
  `apple-developer-id-application` in the Burrow identity key ring. The manual
  `apple-developer-id-kms-csr.yml` workflow uses Authentik-backed Google WIF to
  generate a standard Developer ID Application CSR without exporting private key
  material.
- Treat Developer ID certificate issuance as a Developer website or Xcode step
  until Apple documents App Store Connect API creation for Developer ID
  certificates. The returned `.cer` is public release material; the private key
  remains in Google KMS and must not be converted into `.p12` form.
- The first KMS-backed G2 Developer ID Application certificate is
  `9JKN6HXBHC`, stored at
  `Apple/Certificates/developer-id-application-9JKN6HXBHC.cer`, and expires on
  `2031-06-08`.
- Add a non-exportable Google KMS RSA-2048/SHA-256 key named
  `apple-ios-distribution` in the Burrow identity key ring. The KMS-backed CSR
  can be submitted through the App Store Connect API with
  `Scripts/apple/create-asc-certificate.mjs`.
- The first KMS-backed iOS Distribution certificate is `3G42677598`, stored at
  `Apple/Certificates/ios-distribution-3G42677598.cer`, and expires on
  `2027-06-07`.
- When `BURROW_IOS_DISTRIBUTION_CERTIFICATE_ID` is set, provisioning-profile
  sync must verify App Store profiles reference that certificate. Stale profiles
  with the same name may be deleted and recreated so CI does not silently sign
  against an older distribution identity.
- Apple release signing uses a staged KMS flow because Google publishes the
  Cloud KMS PKCS#11 library for Linux release runners, not macOS runners:
  - macOS Namespace builders compile unsigned iOS device and macOS app bundles
    with the existing Xcode project.
  - Linux release runners download those staged artifacts, authenticate to
    Google through Authentik WIF, verify the Apple `.cer` public key matches the
    intended Google KMS key version, embed the synced provisioning profiles, and
    sign the app/extension bundles with the patched `rcodesign` PKCS#11 path.
  - The signed iOS IPA is uploaded to App Store Connect, and the signed macOS
    ZIP is used as the Sparkle tester artifact.
- Add a Google KMS key named `sparkle-ed25519` for small Sparkle signing probes
  and future signing-service work. It uses `EC_SIGN_ED25519` with software
  protection level because Google KMS rejects Ed25519 for HSM protection. Google
  KMS also rejects normal macOS release archives as direct Ed25519 messages, so
  the tester pipeline signs full-size Sparkle archives directly from the
  decrypted `SPARKLE_EDDSA_KEY_PATH` release seed until a compatible KMS-backed
  Sparkle signing service is introduced. macOS builds embed public EdDSA key
  `uugZuJeqvvKd91NZ6F1Fv2cQenUbIG/ZW3L9MuaEz30=`.
- Use Namespace labels for platform lanes:
  - `namespace-profile-linux-medium`
  - `namespace-profile-macos-large`
  - `namespace-profile-windows-large`
- Use dedicated Apple distribution labels for the focused tester-distribution
  workflow so stale generic Namespace jobs cannot consume Apple release runners:
  - `namespace-profile-linux-medium-apple-v2`
  - `namespace-profile-macos-large-apple-v2`
  Label revisions are allowed when earlier accepted workflows have stale
  queued jobs; the autoscaler should serve only the current focused label set.
- Configure macOS Namespace targets with the Compute API CPU-by-memory shape
  format. `12x28` is the Burrow large macOS baseline; platform-prefixed values
  such as `macos/arm64:8x16` are not accepted by the macOS launcher.
- Treat macOS shape fallback as part of the release lane rather than as a
  best-effort debug path. When large shapes are quota-constrained, the
  dispatcher may try smaller accepted shapes, but every fallback must have a
  bounded create timeout and additional shapes must reuse the final timeout
  policy instead of crashing the dispatcher.
- Keep the shared Linux runner work directory out of macOS bootstraps. The
  dispatcher normalizes macOS runners to `/tmp/forgejo-runner` when the shared
  config points at Linux state paths such as `/var/lib/forgejo-runner`.
- Do not destroy Namespace Linux runners until `nsc describe` shows a real
  terminal container state or tombstone. Empty describe payloads mean the
  environment is still starting, not that the one-job runner has finished.
- Keep the autoscaler-to-dispatcher HTTP timeout longer than the runner TTL.
  Dispatch requests intentionally block until the one-job runner exits, so short
  HTTP client timeouts cancel macOS provisioning mid-bootstrap.
- Run forge Namespace services with the repo-owned `nsc` package and keep it
  current with the token format expected by Namespace. `nsc 0.0.520` is the
  first required baseline for revokable token-file auth. The dispatcher and
  autoscaler systemd units must also use the repo-built packages so runner fixes
  ship with Burrow deploys instead of staying pinned to the module input.
- Render the sealed Namespace runner token with `nsc token create --token_file`
  instead of `nsc auth generate-dev-token`. The dev-token output is for direct
  API/registry access, while the dispatcher needs the JSON token-file format
  consumed by `nsc run`.
- Add a Rust `burrow-windows-stub` binary so the Windows lane can validate packaging and upload mechanics before native Windows networking support is implemented.
- Treat Sparkle as a macOS public-channel upload lane:
  - Generate staged appcast content from the signed macOS artifact when present.
  - Publish `sparkle/<channel>/`, `sparkle/appcast.xml`, and `sparkle/default/` pointers from `publish-store-uploads`.
  - The macOS app embeds Sparkle and consumes the `https://releases.burrow.net/sparkle/appcast.xml` feed.
- Treat TestFlight distribution as a retryable post-upload step. Apple build
  processing can exceed 30 minutes after a successful IPA upload, so
  `apple-distribute-testers` and `publish-store-uploads` wait up to 7200
  seconds by default and allow Forgejo variable
  `TESTFLIGHT_WAIT_PROCESSING_TIMEOUT_SECONDS` to tune that ceiling without a
  repo change. TestFlight wait jobs must use `namespace-profile-linux-testflight`
  or `namespace-profile-macos-large-testflight`; the generic Linux and Apple
  macOS profiles expire before the 7200-second processing window. The
  autoscaler-to-dispatcher timeout must stay above those runner TTLs because
  dispatch requests block until the one-job runner exits. `publish-store-uploads`
  has a separate no-`needs` `distribute-ios-testflight` retry job for
  `distribute_testflight=true` and `upload_app_store=false`, so skipped
  upload/signing jobs cannot suppress distribution of an already-uploaded build.
  `distribute-testflight` is the narrower one-job retry workflow when only App
  Store Connect processing and tester assignment need to be retried.
- Add `infra/releases` as the OpenTofu boundary for the GCS release and package repository backup buckets. Garage is the required S3-compatible primary distribution target after host bootstrap. `infra/identity` remains the boundary for Google KMS signing keys, the Authentik WIF pool/provider, and the runner service account.
- Serve `releases.burrow.net` and `packages.burrow.net` from the forge Caddy
  edge as stable Burrow domains backed by the GCS release and package buckets.
  Appcast enclosure URLs and package repository metadata must use those Burrow
  domains instead of provider-specific object-storage URLs.

## Security and Operational Considerations

- Store credentials are consumed only by signed build or upload jobs, not by ordinary build/test jobs.
- GCS backup and Google KMS access must use Authentik-backed Workload Identity Federation against project `project-88c23ce9-918a-470a-b33`; workflows must not use Google service-account JSON keys.
- Garage uploads use S3-compatible access keys scoped to release/package buckets. They are required by default; GCS is a backup mirror, not the success fallback.
- Garage backup uses a separate read-only key spanning the release, package, and Attic buckets. That key may be sealed into Forgejo/OpenBao for backup jobs, but it must not be reused for release/package writes.
- Attic cache publishing uses a scoped CI push token for the `burrow` cache instead of the bootstrap admin token.
- rclone is not part of the release path. Multi-cloud failover should be added through provider-specific or S3-compatible wrappers with explicit credentials and evidence.
- Required upload credentials are checked explicitly after `./.forgejo/actions/decrypt-release-secrets` materializes them from age:
  - `secrets/apple/appstore-connect.env.age`: `ASC_API_KEY_ID`, `ASC_API_ISSUER_ID`, and base64-encoded `.p8` key material.
  - KMS-backed Apple signing must not use `.p12` private-key material. The
    legacy `secrets/apple/distribution-signing.env.age` path remains accepted
    only for emergency keychain signing and should not be used for the normal
    tester release lane.
  - Sparkle appcasts are signed with Google KMS by default; the legacy
    `secrets/apple/sparkle.env.age` path remains accepted only for emergency
    local EdDSA signing.
  - future Microsoft Partner Center credentials
- The temporary CI keychain password is always the distribution certificate password; there is no separate keychain-password release secret.
- KMS-backed Apple certificates do not satisfy the existing Xcode/keychain
  import path. Shipping with these certificates requires a signer that can
  delegate Apple code-signing operations to Google KMS.
- If the first key ring or Developer ID key is bootstrapped before the
  OpenTofu remote state backend is available locally, the live resources must be
  imported into `infra/identity` before enabling managed identity applies.
- Dedicated store accounts should be sealed with agenix before enabling always-on uploads. The runner only needs an age identity such as `/var/lib/forgejo-runner-agent/age_keystore`; `AGE_FORGE_SSH_KEY` is a fallback bootstrap secret, not a place for release material.
- The canonical identity registry reserves `release@burrow.net` for release automation and `namespace@burrow.net` for Namespace runner capacity credentials; secrets stay outside `contributors.nix`.
- Grafana starts with a sealed admin password and a pending OIDC client secret. Local admin login remains available until the real OIDC secret is rotated and the Authentik reconciliation service creates the Grafana provider.
- OpenTofu apply/import modes require an explicit confirmation and remote state. Validation may run without remote state.
- Build tags are pushed only after artifacts have been assembled for the build number.
- External TestFlight distribution and public app-store rollout must stay explicit; internal or private lanes can be defaulted only after credentials and metadata are verified.

## Contributor Playbook

1. Check whether a release is needed:
   ```bash
   Scripts/version.sh status
   ```
2. Build local artifacts:
   ```bash
   BUILD_NUMBER=local Scripts/ci/build-release-artifacts.sh all
   ```
3. Build Apple release validation artifacts on macOS:
   ```bash
   BUILD_NUMBER=local bazel build //bazel/apple:release_ios_stamp //bazel/apple:release_macos_stamp
   ```
4. Verify the Windows stub still builds:
   ```bash
   cargo test --locked -p burrow-windows-stub
   ```
5. Validate BEP metadata:
   ```bash
   python3 Scripts/check-bep-metadata.py
   ```
6. Before turning on an upload lane, seal the dedicated service-account secret and run:
   ```bash
   Scripts/seal-release-secrets.sh --source-dir intake/release
   Scripts/ci/check-release-config.sh app-store
   Scripts/ci/check-release-config.sh apple-signing
   Scripts/ci/check-release-config.sh gcs
   ```
7. Treat successful Forgejo release asset publication and successful store upload jobs as separate evidence points.
8. Validate Grafana OpenTofu syntax locally:
   ```bash
   Scripts/grafana-tofu.sh init -backend=false
   Scripts/grafana-tofu.sh validate
   ```
9. Validate release storage OpenTofu syntax locally:
   ```bash
   Scripts/releases-tofu.sh init -backend=false
   Scripts/releases-tofu.sh validate
   ```
10. Validate release storage wrapper syntax:
   ```bash
   bash -n Scripts/ci/upload-release-storage.sh Scripts/ci/upload-package-storage.sh Scripts/ci/backup-garage-to-gcs.sh Scripts/ci/publish-nix-cache.sh
   ```
11. Dispatch a Forgejo workflow without GitHub:
   ```bash
   Scripts/forgejo-dispatch.sh build-release.yml --ref main
   ```
12. After deploying the forge host, seal:
    ```text
    BURROW_NIX_CACHE_PUSH_TOKEN from /var/lib/burrow/nix-cache/ci-push-token
    BURROW_GARAGE_BACKUP_ACCESS_KEY_ID from /var/lib/burrow/garage/env
    BURROW_GARAGE_BACKUP_SECRET_ACCESS_KEY from /var/lib/burrow/garage/env
    ```

## Alternatives Considered

- Keep one monolithic release workflow. Rejected because build failures, upload failures, and credential gaps become hard to distinguish.
- Wait for native Windows support before adding a Windows lane. Rejected because packaging and Microsoft Store authentication can be validated with a Rust stub first.
- Add Sparkle to the Apple app in the same change. Rejected because Sparkle changes the client update trust boundary and should be reviewed separately from release plumbing.

## Impact on Other Work

- Future Apple release work can fill in signed archive/export details without changing release numbering.
- Future Windows work can replace the stub package with the native client while preserving upload shape.
- Identity and credential follow-up should derive long-lived automation accounts from `contributors.nix` where they represent Burrow-operated identities.

## Decision

Pending.

## References

- `CONSTITUTION.md`
- `contributors.nix`
- `.forgejo/workflows/release-if-needed.yml`
- `.forgejo/workflows/build-release.yml`
- `.forgejo/workflows/publish-store-uploads.yml`
- `Scripts/version.sh`
- `Scripts/ci/nscloud-cache.sh`
- `Scripts/ci/ensure-nix.sh`
- `Scripts/ci/package-apple-artifacts.sh`
- `Scripts/ci/upload-release-storage.sh`
- `Scripts/ci/upload-package-storage.sh`
- `Scripts/ci/publish-nix-cache.sh`
- `Scripts/ci/backup-garage-to-gcs.sh`
- `bazel/apple/BUILD.bazel`
- `Scripts/forgejo-dispatch.sh`
- `Scripts/forgejo-dispatch-via-host.sh`
- `Scripts/grafana-tofu.sh`
- `Scripts/releases-tofu.sh`
- `.forgejo/workflows/infra-opentofu.yml`
- `.forgejo/workflows/publish-nix-cache.yml`
- `.forgejo/workflows/backup-garage-storage.yml`
- `infra/grafana/`
- `infra/releases/`
- `nixos/modules/burrow-garage.nix`
- `nixos/modules/burrow-observability.nix`
- `services/grafana/dashboards/burrow-overview.json`
- `services/grafana/dashboards/headscale.json`
- `services/grafana/dashboards/observability.json`
