# `BEP-0009` - Release Infrastructure and Store Upload Pipeline

```text
Status: Draft
Proposal: BEP-0009
Authors: gpt-5.5
Coordinator: gpt-5.5
Reviewers: Pending
Constitution Sections: II, III, IV, V
Implementation PRs: Pending
Decision Date: Pending
```

## Summary

Burrow needs a release pipeline that behaves like the project infrastructure it is replacing: release eligibility is detected from forge build tags, build artifacts are produced per platform, upload lanes are separate from build lanes, and store credentials are explicit automation inputs rather than hidden local state.

The first implementation creates the release spine now: `release-if-needed` dispatches a build when HEAD has not been tagged as released, `build-release` packages current artifacts on Namespace-backed runners, and `publish-store-uploads` performs guarded upload steps for App Store Connect, Sparkle, Microsoft Store, and storage-backed public channels after Authentik-backed Google WIF authentication.

The same change also makes the forge itself the release authority: release tags prefer the Burrow Forgejo remote, dispatch helpers talk to Forgejo directly with a live-host fallback, runner identity is discoverable from persistent host state, repository mirror sync is disabled for the canonical Burrow repository, `nix.burrow.net` provides a Burrow-owned Nix cache for workers, and Grafana observability is bootstrapped with Nix-owned Prometheus/OpenTelemetry/Jaeger service state plus OpenTofu-owned dashboards.

## Motivation

- Burrow already controls its forge and runner infrastructure, but the release path only produced a Linux CLI tarball.
- Apple, Linux, Sparkle, and future Windows outputs need the same build-number and artifact layout so release evidence is inspectable.
- The Namespace account should be part of the live runner surface, including macOS and Windows lanes, instead of sitting idle behind unused configuration.
- Authentication and store credentials should be named and checked at lane boundaries so dedicated accounts can be added without rewriting workflows.

## Detailed Design

- Use lightweight `builds/<number>` tags as the authoritative release watermark.
- Keep `Scripts/version.sh status` as the release-needed check:
  - `clean` means HEAD is exactly the latest authoritative build tag.
  - `dirty` means the release workflow should create a new build number.
- Split workflows:
  - `.forgejo/workflows/release-if-needed.yml` checks release status and dispatches `build-release.yml`.
  - `.forgejo/workflows/build-release.yml` builds and stages artifacts.
  - `.forgejo/workflows/publish-store-uploads.yml` uploads staged artifacts to external stores and public channels.
- Stage artifacts under `dist/builds/<build-number>/<platform>/`.
- Publish generic release assets to a Forgejo release named `Build <number>`.
- Upload release artifacts through `Scripts/ci/upload-release-storage.sh`. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the `burrow-net-releases` GCS backup bucket unless `BURROW_RELEASE_GCS_BACKUP=false`.
- Publish signed package repository trees through `Scripts/ci/upload-package-storage.sh`. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the `burrow-net-packages` GCS backup bucket unless `BURROW_PACKAGE_GCS_BACKUP=false`.
- Publish selected Nix closures through `.forgejo/workflows/publish-nix-cache.yml` and `Scripts/ci/publish-nix-cache.sh`. The workflow logs into `nix.burrow.net` with the scoped `BURROW_NIX_CACHE_PUSH_TOKEN` generated by host bootstrap and skips when the token is absent unless explicitly required.
- Mirror Garage buckets through `.forgejo/workflows/backup-garage-storage.yml` and `Scripts/ci/backup-garage-to-gcs.sh`. The scheduled workflow uses the host-generated Garage read-only backup key, Authentik-backed Google WIF, and `gcloud storage rsync` to back up `burrow-releases`, `burrow-packages`, and `attic` to GCS.
- Build Apple artifacts on `namespace-profile-macos-large` through Bazel wrapper targets:
  - `//bazel/apple:release_ios_stamp`
  - `//bazel/apple:release_macos_stamp`
  These targets wrap the existing Xcode project and give the workflow one stable entrypoint that shares the same Bazel/Nix cache setup while Burrow decides whether deeper `rules_apple` targets are worth the migration.
- Use Namespace cache setup before Bazel/Nix work. Prefer the Namespace Bazel remote cache when `nsc` is authenticated and fall back to a local disk/repository cache when remote setup is unavailable.
- Produce unsigned Apple validation artifacts when signing is absent, but require signing for an App Store requested iOS release and for a Sparkle tester release. Uploadable artifacts are produced only after provisioning profiles can be synced from App Store Connect credentials and the relevant Apple certificate is proven to match the selected Google KMS key.
- Resolve release credentials through age/agenix before signed Apple lanes run. Forgejo secrets should carry only the runner age identity fallback and the Authentik WIF client secret until OpenBao can mint the runner token directly. App Store Connect keys and signing certificates live as sealed files under `secrets/`.
- Persist the forge runner SSH key as `/var/lib/forgejo-runner-agent/age_keystore` and export `BURROW_RUNNER_AGE_IDENTITY_PATH` so jobs can resolve agenix identities without embedding long-lived material in every workflow.
- Add `Scripts/forgejo-dispatch.sh` and `Scripts/forgejo-dispatch-via-host.sh`:
  - local dispatch uses an explicit token, local intake token, or the sealed forgejo-nsc dispatcher config when available;
  - host fallback SSHes to the live forge and reads the agenix-materialized dispatcher config there;
  - neither path relies on a GitHub remote or GitHub workflow API.
- Default build tag reads and pushes to the Burrow Forgejo remote when that remote exists. `BURROW_VERSION_REMOTE` remains the override for unusual migrations.
- Keep the canonical Forgejo repo as an ordinary source repository, not a mirror. The host config disables new mirror creation and removes mirror/push-mirror rows for the canonical Burrow repository at activation time.
- Add `.forgejo/workflows/infra-opentofu.yml` and `Scripts/grafana-tofu.sh` for Grafana API-managed folders and dashboards. NixOS owns the Grafana service, SSO, reverse proxy, datasources, Prometheus, OpenTelemetry collector, Jaeger, secrets, and boot-time configuration; OpenTofu owns API-created dashboard objects.
- Sync provisioning profiles from App Store Connect in CI using the decrypted API key:
  - iOS App Store profiles for `net.burrow.app` and `net.burrow.app.network`.
  - macOS Developer ID profiles for the same bundle identifiers.
  Provisioning profile `.age` files are not part of the normal release path.
- Add a non-exportable Google KMS RSA-2048/SHA-256 key named
  `apple-developer-id-application` in the Burrow identity key ring. The manual
  `apple-developer-id-kms-csr.yml` workflow uses Authentik-backed Google WIF to
  generate a standard Developer ID Application CSR without exporting private key
  material.
- Treat Developer ID certificate issuance as a Developer website or Xcode step
  until Apple documents App Store Connect API creation for Developer ID
  certificates. The returned `.cer` is public release material; the private key
  remains in Google KMS and must not be converted into `.p12` form.
- The first KMS-backed G2 Developer ID Application certificate is
  `9JKN6HXBHC`, stored at
  `Apple/Certificates/developer-id-application-9JKN6HXBHC.cer`, and expires on
  `2031-06-08`.
- Add a non-exportable Google KMS RSA-2048/SHA-256 key named
  `apple-ios-distribution` in the Burrow identity key ring. The KMS-backed CSR
  can be submitted through the App Store Connect API with
  `Scripts/apple/create-asc-certificate.mjs`.
- The first KMS-backed iOS Distribution certificate is `3G42677598`, stored at
  `Apple/Certificates/ios-distribution-3G42677598.cer`, and expires on
  `2027-06-07`.
- When `BURROW_IOS_DISTRIBUTION_CERTIFICATE_ID` is set, provisioning-profile
  sync must verify App Store profiles reference that certificate. Stale profiles
  with the same name may be deleted and recreated so CI does not silently sign
  against an older distribution identity.
- Apple release signing uses a staged KMS flow because Google publishes the
  Cloud KMS PKCS#11 library for Linux release runners, not macOS runners:
  - macOS Namespace builders compile unsigned iOS device and macOS app bundles
    with the existing Xcode project.
  - Linux release runners download those staged artifacts, authenticate to
    Google through Authentik WIF, verify the Apple `.cer` public key matches the
    intended Google KMS key version, embed the synced provisioning profiles, and
    sign the app/extension bundles with the patched `rcodesign` PKCS#11 path.
  - The signed iOS IPA is uploaded to App Store Connect, and the signed macOS
    ZIP is used as the Sparkle tester artifact.
- Add a non-exportable Google KMS key named `sparkle-ed25519` for Sparkle
  appcast signatures. It uses `EC_SIGN_ED25519` with software protection level
  because Google KMS rejects Ed25519 for HSM protection. macOS builds embed
  public EdDSA key `Myv9ZNZT6YGKMtMezh52ra4WqaeEKc4VlvVU0evhJeI=`.
- Use Namespace labels for platform lanes:
  - `namespace-profile-linux-medium`
  - `namespace-profile-macos-large`
  - `namespace-profile-windows-large`
- Use dedicated Apple distribution labels for the focused tester-distribution
  workflow so stale generic Namespace jobs cannot consume Apple release runners:
  - `namespace-profile-linux-medium-apple`
  - `namespace-profile-macos-large-apple`
- Configure macOS Namespace targets with the Compute API CPU-by-memory shape
  format. `12x28` is the Burrow large macOS baseline; platform-prefixed values
  such as `macos/arm64:8x16` are not accepted by the macOS launcher.
- Keep the shared Linux runner work directory out of macOS bootstraps. The
  dispatcher normalizes macOS runners to `/tmp/forgejo-runner` when the shared
  config points at Linux state paths such as `/var/lib/forgejo-runner`.
- Do not destroy Namespace Linux runners until `nsc describe` shows a real
  terminal container state or tombstone. Empty describe payloads mean the
  environment is still starting, not that the one-job runner has finished.
- Run forge Namespace services with the repo-owned `nsc` package and keep it
  current with the token format expected by Namespace. `nsc 0.0.520` is the
  first required baseline for revokable token-file auth. The dispatcher and
  autoscaler systemd units must also use the repo-built packages so runner fixes
  ship with Burrow deploys instead of staying pinned to the module input.
- Render the sealed Namespace runner token with `nsc token create --token_file`
  instead of `nsc auth generate-dev-token`. The dev-token output is for direct
  API/registry access, while the dispatcher needs the JSON token-file format
  consumed by `nsc run`.
- Add a Rust `burrow-windows-stub` binary so the Windows lane can validate packaging and upload mechanics before native Windows networking support is implemented.
- Treat Sparkle as a macOS public-channel upload lane:
  - Generate staged appcast content from the signed macOS artifact when present.
  - Publish `sparkle/<channel>/`, `sparkle/appcast.xml`, and `sparkle/default/` pointers from `publish-store-uploads`.
  - The macOS app embeds Sparkle and consumes the `https://releases.burrow.net/sparkle/appcast.xml` feed.
- Add `infra/releases` as the OpenTofu boundary for the GCS release and package repository backup buckets. Garage is the required S3-compatible primary distribution target after host bootstrap. `infra/identity` remains the boundary for Google KMS signing keys, the Authentik WIF pool/provider, and the runner service account.

## Security and Operational Considerations

- Store credentials are consumed only by signed build or upload jobs, not by ordinary build/test jobs.
- GCS backup and Google KMS access must use Authentik-backed Workload Identity Federation against project `project-88c23ce9-918a-470a-b33`; workflows must not use Google service-account JSON keys.
- Garage uploads use S3-compatible access keys scoped to release/package buckets. They are required by default; GCS is a backup mirror, not the success fallback.
- Garage backup uses a separate read-only key spanning the release, package, and Attic buckets. That key may be sealed into Forgejo/OpenBao for backup jobs, but it must not be reused for release/package writes.
- Attic cache publishing uses a scoped CI push token for the `burrow` cache instead of the bootstrap admin token.
- rclone is not part of the release path. Multi-cloud failover should be added through provider-specific or S3-compatible wrappers with explicit credentials and evidence.
- Required upload credentials are checked explicitly after `./.forgejo/actions/decrypt-release-secrets` materializes them from age:
  - `secrets/apple/appstore-connect.env.age`: `ASC_API_KEY_ID`, `ASC_API_ISSUER_ID`, and base64-encoded `.p8` key material.
  - KMS-backed Apple signing must not use `.p12` private-key material. The
    legacy `secrets/apple/distribution-signing.env.age` path remains accepted
    only for emergency keychain signing and should not be used for the normal
    tester release lane.
  - Sparkle appcasts are signed with Google KMS by default; the legacy
    `secrets/apple/sparkle.env.age` path remains accepted only for emergency
    local EdDSA signing.
  - future Microsoft Partner Center credentials
- The temporary CI keychain password is always the distribution certificate password; there is no separate keychain-password release secret.
- KMS-backed Apple certificates do not satisfy the existing Xcode/keychain
  import path. Shipping with these certificates requires a signer that can
  delegate Apple code-signing operations to Google KMS.
- If the first key ring or Developer ID key is bootstrapped before the
  OpenTofu remote state backend is available locally, the live resources must be
  imported into `infra/identity` before enabling managed identity applies.
- Dedicated store accounts should be sealed with agenix before enabling always-on uploads. The runner only needs an age identity such as `/var/lib/forgejo-runner-agent/age_keystore`; `AGE_FORGE_SSH_KEY` is a fallback bootstrap secret, not a place for release material.
- The canonical identity registry reserves `release@burrow.net` for release automation and `namespace@burrow.net` for Namespace runner capacity credentials; secrets stay outside `contributors.nix`.
- Grafana starts with a sealed admin password and a pending OIDC client secret. Local admin login remains available until the real OIDC secret is rotated and the Authentik reconciliation service creates the Grafana provider.
- OpenTofu apply/import modes require an explicit confirmation and remote state. Validation may run without remote state.
- Build tags are pushed only after artifacts have been assembled for the build number.
- External TestFlight distribution and public app-store rollout must stay explicit; internal or private lanes can be defaulted only after credentials and metadata are verified.

## Contributor Playbook

1. Check whether a release is needed:
   ```bash
   Scripts/version.sh status
   ```
2. Build local artifacts:
   ```bash
   BUILD_NUMBER=local Scripts/ci/build-release-artifacts.sh all
   ```
3. Build Apple release validation artifacts on macOS:
   ```bash
   BUILD_NUMBER=local bazel build //bazel/apple:release_ios_stamp //bazel/apple:release_macos_stamp
   ```
4. Verify the Windows stub still builds:
   ```bash
   cargo test --locked -p burrow-windows-stub
   ```
5. Validate BEP metadata:
   ```bash
   python3 Scripts/check-bep-metadata.py
   ```
6. Before turning on an upload lane, seal the dedicated service-account secret and run:
   ```bash
   Scripts/seal-release-secrets.sh --source-dir intake/release
   Scripts/ci/check-release-config.sh app-store
   Scripts/ci/check-release-config.sh apple-signing
   Scripts/ci/check-release-config.sh gcs
   ```
7. Treat successful Forgejo release asset publication and successful store upload jobs as separate evidence points.
8. Validate Grafana OpenTofu syntax locally:
   ```bash
   Scripts/grafana-tofu.sh init -backend=false
   Scripts/grafana-tofu.sh validate
   ```
9. Validate release storage OpenTofu syntax locally:
   ```bash
   Scripts/releases-tofu.sh init -backend=false
   Scripts/releases-tofu.sh validate
   ```
10. Validate release storage wrapper syntax:
   ```bash
   bash -n Scripts/ci/upload-release-storage.sh Scripts/ci/upload-package-storage.sh Scripts/ci/backup-garage-to-gcs.sh Scripts/ci/publish-nix-cache.sh
   ```
11. Dispatch a Forgejo workflow without GitHub:
   ```bash
   Scripts/forgejo-dispatch.sh build-release.yml --ref main
   ```
12. After deploying the forge host, seal:
    ```text
    BURROW_NIX_CACHE_PUSH_TOKEN from /var/lib/burrow/nix-cache/ci-push-token
    BURROW_GARAGE_BACKUP_ACCESS_KEY_ID from /var/lib/burrow/garage/env
    BURROW_GARAGE_BACKUP_SECRET_ACCESS_KEY from /var/lib/burrow/garage/env
    ```

## Alternatives Considered

- Keep one monolithic release workflow. Rejected because build failures, upload failures, and credential gaps become hard to distinguish.
- Wait for native Windows support before adding a Windows lane. Rejected because packaging and Microsoft Store authentication can be validated with a Rust stub first.
- Add Sparkle to the Apple app in the same change. Rejected because Sparkle changes the client update trust boundary and should be reviewed separately from release plumbing.

## Impact on Other Work

- Future Apple release work can fill in signed archive/export details without changing release numbering.
- Future Windows work can replace the stub package with the native client while preserving upload shape.
- Identity and credential follow-up should derive long-lived automation accounts from `contributors.nix` where they represent Burrow-operated identities.

## Decision

Pending.

## References

- `CONSTITUTION.md`
- `contributors.nix`
- `.forgejo/workflows/release-if-needed.yml`
- `.forgejo/workflows/build-release.yml`
- `.forgejo/workflows/publish-store-uploads.yml`
- `Scripts/version.sh`
- `Scripts/ci/nscloud-cache.sh`
- `Scripts/ci/ensure-nix.sh`
- `Scripts/ci/package-apple-artifacts.sh`
- `Scripts/ci/upload-release-storage.sh`
- `Scripts/ci/upload-package-storage.sh`
- `Scripts/ci/publish-nix-cache.sh`
- `Scripts/ci/backup-garage-to-gcs.sh`
- `bazel/apple/BUILD.bazel`
- `Scripts/forgejo-dispatch.sh`
- `Scripts/forgejo-dispatch-via-host.sh`
- `Scripts/grafana-tofu.sh`
- `Scripts/releases-tofu.sh`
- `.forgejo/workflows/infra-opentofu.yml`
- `.forgejo/workflows/publish-nix-cache.yml`
- `.forgejo/workflows/backup-garage-storage.yml`
- `infra/grafana/`
- `infra/releases/`
- `nixos/modules/burrow-garage.nix`
- `nixos/modules/burrow-observability.nix`
- `services/grafana/dashboards/burrow-overview.json`
- `services/grafana/dashboards/headscale.json`
- `services/grafana/dashboards/observability.json`