burrow/evolution/proposals/BEP-0009-release-infrastructure-and-store-upload-pipeline.md

BEP-0009 - Release Infrastructure and Store Upload Pipeline

Status: Draft
Proposal: BEP-0009
Authors: gpt-5.5
Coordinator: gpt-5.5
Reviewers: Pending
Constitution Sections: II, III, IV, V
Implementation PRs: Pending
Decision Date: Pending

Summary

Burrow needs a release pipeline that behaves like the project infrastructure it is replacing: release eligibility is detected from forge build tags, build artifacts are produced per platform, upload lanes are separate from build lanes, and store credentials are explicit automation inputs rather than hidden local state.

The first implementation creates the release spine now: release-if-needed dispatches a build when HEAD has not been tagged as released, build-release packages current artifacts on Namespace-backed runners, and publish-store-uploads performs guarded upload steps for App Store Connect, Sparkle, Microsoft Store, and storage-backed public channels after Authentik-backed Google WIF authentication.

The same change also makes the forge itself the release authority: release tags prefer the Burrow Forgejo remote, dispatch helpers talk to Forgejo directly with a live-host fallback, runner identity is discoverable from persistent host state, repository mirror sync is disabled for the canonical Burrow repository, nix.burrow.net provides a Burrow-owned Nix cache for workers, and Grafana observability is bootstrapped with Nix-owned Prometheus/OpenTelemetry/Jaeger service state plus OpenTofu-owned dashboards.

Motivation

• Burrow already controls its forge and runner infrastructure, but the release path only produced a Linux CLI tarball.
• Apple, Linux, Sparkle, and future Windows outputs need the same build-number and artifact layout so release evidence is inspectable.
• The Namespace account should be part of the live runner surface, including macOS and Windows lanes, instead of sitting idle behind unused configuration.
• Authentication and store credentials should be named and checked at lane boundaries so dedicated accounts can be added without rewriting workflows.

Detailed Design

• Use lightweight builds/<number> tags as the authoritative release watermark.
• Keep Scripts/version.sh status as the release-needed check:
  • clean means HEAD is exactly the latest authoritative build tag.
  • dirty means the release workflow should create a new build number.
• Split workflows:
  • .forgejo/workflows/release-if-needed.yml checks release status and dispatches build-release.yml.
  • .forgejo/workflows/build-release.yml builds and stages artifacts.
  • .forgejo/workflows/publish-store-uploads.yml uploads staged artifacts to external stores and public channels.
• Stage artifacts under dist/builds/<build-number>/<platform>/.
• Publish generic release assets to a Forgejo release named Build <number>.
• Upload release artifacts through Scripts/ci/upload-release-storage.sh. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the burrow-net-releases GCS backup bucket unless BURROW_RELEASE_GCS_BACKUP=false.
• Publish signed package repository trees through Scripts/ci/upload-package-storage.sh. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the burrow-net-packages GCS backup bucket unless BURROW_PACKAGE_GCS_BACKUP=false.
• Publish selected Nix closures through .forgejo/workflows/publish-nix-cache.yml and Scripts/ci/publish-nix-cache.sh. The workflow logs into nix.burrow.net with the scoped BURROW_NIX_CACHE_PUSH_TOKEN generated by host bootstrap and skips when the token is absent unless explicitly required.
• Mirror Garage buckets through .forgejo/workflows/backup-garage-storage.yml and Scripts/ci/backup-garage-to-gcs.sh. The scheduled workflow uses the host-generated Garage read-only backup key, Authentik-backed Google WIF, and gcloud storage rsync to back up burrow-releases, burrow-packages, and attic to GCS.
• Build Apple artifacts on namespace-profile-macos-large through Bazel wrapper targets:
  • //bazel/apple:release_ios_stamp
  • //bazel/apple:release_macos_stamp These targets wrap the existing Xcode project and give the workflow one stable entrypoint that shares the same Bazel/Nix cache setup while Burrow decides whether deeper rules_apple targets are worth the migration.
• Use Namespace cache setup before Bazel/Nix work. Prefer the Namespace Bazel remote cache when nsc is authenticated and fall back to a local disk/repository cache when remote setup is unavailable.
• Produce unsigned Apple validation artifacts when signing is absent, but require signing for an App Store requested iOS release and for a Sparkle tester release. Uploadable artifacts are produced only after provisioning profiles can be synced from App Store Connect credentials and the relevant Apple certificate is proven to match the selected Google KMS key.
• Resolve release credentials through age/agenix before signed Apple lanes run. Forgejo secrets should carry only the runner age identity fallback and the Authentik WIF client secret until OpenBao can mint the runner token directly. App Store Connect keys and signing certificates live as sealed files under secrets/.
• Persist the forge runner SSH key as /var/lib/forgejo-runner-agent/age_keystore and export BURROW_RUNNER_AGE_IDENTITY_PATH so jobs can resolve agenix identities without embedding long-lived material in every workflow.
• Add Scripts/forgejo-dispatch.sh and Scripts/forgejo-dispatch-via-host.sh:
  • local dispatch uses an explicit token, local intake token, or the sealed forgejo-nsc dispatcher config when available;
  • host fallback SSHes to the live forge and reads the agenix-materialized dispatcher config there;
  • neither path relies on a GitHub remote or GitHub workflow API.
• Default build tag reads and pushes to the Burrow Forgejo remote when that remote exists. BURROW_VERSION_REMOTE remains the override for unusual migrations.
• Keep the canonical Forgejo repo as an ordinary source repository, not a mirror. The host config disables new mirror creation and removes mirror/push-mirror rows for the canonical Burrow repository at activation time.
• Add .forgejo/workflows/infra-opentofu.yml and Scripts/grafana-tofu.sh for Grafana API-managed folders and dashboards. NixOS owns the Grafana service, SSO, reverse proxy, datasources, Prometheus, OpenTelemetry collector, Jaeger, secrets, and boot-time configuration; OpenTofu owns API-created dashboard objects.
• Sync provisioning profiles from App Store Connect in CI using the decrypted API key:
  • iOS App Store profiles for net.burrow.app and net.burrow.app.network.
  • macOS Developer ID profiles for the same bundle identifiers. Provisioning profile .age files are not part of the normal release path.
• Add a non-exportable Google KMS RSA-2048/SHA-256 key named apple-developer-id-application in the Burrow identity key ring. The manual apple-developer-id-kms-csr.yml workflow uses Authentik-backed Google WIF to generate a standard Developer ID Application CSR without exporting private key material.
• Treat Developer ID certificate issuance as a Developer website or Xcode step until Apple documents App Store Connect API creation for Developer ID certificates. The returned .cer is public release material; the private key remains in Google KMS and must not be converted into .p12 form.
• The first KMS-backed G2 Developer ID Application certificate is 9JKN6HXBHC, stored at Apple/Certificates/developer-id-application-9JKN6HXBHC.cer, and expires on 2031-06-08.
• Add a non-exportable Google KMS RSA-2048/SHA-256 key named apple-ios-distribution in the Burrow identity key ring. The KMS-backed CSR can be submitted through the App Store Connect API with Scripts/apple/create-asc-certificate.mjs.
• The first KMS-backed iOS Distribution certificate is 3G42677598, stored at Apple/Certificates/ios-distribution-3G42677598.cer, and expires on 2027-06-07.
• When BURROW_IOS_DISTRIBUTION_CERTIFICATE_ID is set, provisioning-profile sync must verify App Store profiles reference that certificate. Stale profiles with the same name may be deleted and recreated so CI does not silently sign against an older distribution identity.
• Apple release signing uses a staged KMS flow because Google publishes the Cloud KMS PKCS#11 library for Linux release runners, not macOS runners:
  • macOS Namespace builders compile unsigned iOS device and macOS app bundles with the existing Xcode project.
  • Linux release runners download those staged artifacts, authenticate to Google through Authentik WIF, verify the Apple .cer public key matches the intended Google KMS key version, embed the synced provisioning profiles, and sign the app/extension bundles with the patched rcodesign PKCS#11 path.
  • The signed iOS IPA is uploaded to App Store Connect, and the signed macOS ZIP is used as the Sparkle tester artifact.
• Add a non-exportable Google KMS key named sparkle-ed25519 for Sparkle appcast signatures. It uses EC_SIGN_ED25519 with software protection level because Google KMS rejects Ed25519 for HSM protection. macOS builds embed public EdDSA key Myv9ZNZT6YGKMtMezh52ra4WqaeEKc4VlvVU0evhJeI=.
• Use Namespace labels for platform lanes:
  • namespace-profile-linux-medium
  • namespace-profile-macos-large
  • namespace-profile-windows-large
• Use dedicated Apple distribution labels for the focused tester-distribution workflow so stale generic Namespace jobs cannot consume Apple release runners:
  • namespace-profile-linux-medium-apple-v2
  • namespace-profile-macos-large-apple-v2 Label revisions are allowed when earlier accepted workflows have stale queued jobs; the autoscaler should serve only the current focused label set.
• Configure macOS Namespace targets with the Compute API CPU-by-memory shape format. 12x28 is the Burrow large macOS baseline; platform-prefixed values such as macos/arm64:8x16 are not accepted by the macOS launcher.
• Keep the shared Linux runner work directory out of macOS bootstraps. The dispatcher normalizes macOS runners to /tmp/forgejo-runner when the shared config points at Linux state paths such as /var/lib/forgejo-runner.
• Do not destroy Namespace Linux runners until nsc describe shows a real terminal container state or tombstone. Empty describe payloads mean the environment is still starting, not that the one-job runner has finished.
• Keep the autoscaler-to-dispatcher HTTP timeout longer than the runner TTL. Dispatch requests intentionally block until the one-job runner exits, so short HTTP client timeouts cancel macOS provisioning mid-bootstrap.
• Run forge Namespace services with the repo-owned nsc package and keep it current with the token format expected by Namespace. nsc 0.0.520 is the first required baseline for revokable token-file auth. The dispatcher and autoscaler systemd units must also use the repo-built packages so runner fixes ship with Burrow deploys instead of staying pinned to the module input.
• Render the sealed Namespace runner token with nsc token create --token_file instead of nsc auth generate-dev-token. The dev-token output is for direct API/registry access, while the dispatcher needs the JSON token-file format consumed by nsc run.
• Add a Rust burrow-windows-stub binary so the Windows lane can validate packaging and upload mechanics before native Windows networking support is implemented.
• Treat Sparkle as a macOS public-channel upload lane:
  • Generate staged appcast content from the signed macOS artifact when present.
  • Publish sparkle/<channel>/, sparkle/appcast.xml, and sparkle/default/ pointers from publish-store-uploads.
  • The macOS app embeds Sparkle and consumes the https://releases.burrow.net/sparkle/appcast.xml feed.
• Add infra/releases as the OpenTofu boundary for the GCS release and package repository backup buckets. Garage is the required S3-compatible primary distribution target after host bootstrap. infra/identity remains the boundary for Google KMS signing keys, the Authentik WIF pool/provider, and the runner service account.

Security and Operational Considerations

• Store credentials are consumed only by signed build or upload jobs, not by ordinary build/test jobs.
• GCS backup and Google KMS access must use Authentik-backed Workload Identity Federation against project project-88c23ce9-918a-470a-b33; workflows must not use Google service-account JSON keys.
• Garage uploads use S3-compatible access keys scoped to release/package buckets. They are required by default; GCS is a backup mirror, not the success fallback.
• Garage backup uses a separate read-only key spanning the release, package, and Attic buckets. That key may be sealed into Forgejo/OpenBao for backup jobs, but it must not be reused for release/package writes.
• Attic cache publishing uses a scoped CI push token for the burrow cache instead of the bootstrap admin token.
• rclone is not part of the release path. Multi-cloud failover should be added through provider-specific or S3-compatible wrappers with explicit credentials and evidence.
• Required upload credentials are checked explicitly after ./.forgejo/actions/decrypt-release-secrets materializes them from age:
  • secrets/apple/appstore-connect.env.age: ASC_API_KEY_ID, ASC_API_ISSUER_ID, and base64-encoded .p8 key material.
  • KMS-backed Apple signing must not use .p12 private-key material. The legacy secrets/apple/distribution-signing.env.age path remains accepted only for emergency keychain signing and should not be used for the normal tester release lane.
  • Sparkle appcasts are signed with Google KMS by default; the legacy secrets/apple/sparkle.env.age path remains accepted only for emergency local EdDSA signing.
  • future Microsoft Partner Center credentials
• The temporary CI keychain password is always the distribution certificate password; there is no separate keychain-password release secret.
• KMS-backed Apple certificates do not satisfy the existing Xcode/keychain import path. Shipping with these certificates requires a signer that can delegate Apple code-signing operations to Google KMS.
• If the first key ring or Developer ID key is bootstrapped before the OpenTofu remote state backend is available locally, the live resources must be imported into infra/identity before enabling managed identity applies.
• Dedicated store accounts should be sealed with agenix before enabling always-on uploads. The runner only needs an age identity such as /var/lib/forgejo-runner-agent/age_keystore; AGE_FORGE_SSH_KEY is a fallback bootstrap secret, not a place for release material.
• The canonical identity registry reserves release@burrow.net for release automation and namespace@burrow.net for Namespace runner capacity credentials; secrets stay outside contributors.nix.
• Grafana starts with a sealed admin password and a pending OIDC client secret. Local admin login remains available until the real OIDC secret is rotated and the Authentik reconciliation service creates the Grafana provider.
• OpenTofu apply/import modes require an explicit confirmation and remote state. Validation may run without remote state.
• Build tags are pushed only after artifacts have been assembled for the build number.
• External TestFlight distribution and public app-store rollout must stay explicit; internal or private lanes can be defaulted only after credentials and metadata are verified.

Contributor Playbook

1. Check whether a release is needed:

   Scripts/version.sh status
   

2. Build local artifacts:

   BUILD_NUMBER=local Scripts/ci/build-release-artifacts.sh all
   

3. Build Apple release validation artifacts on macOS:

   BUILD_NUMBER=local bazel build //bazel/apple:release_ios_stamp //bazel/apple:release_macos_stamp
   

4. Verify the Windows stub still builds:

   cargo test --locked -p burrow-windows-stub
   

5. Validate BEP metadata:

   python3 Scripts/check-bep-metadata.py
   

6. Before turning on an upload lane, seal the dedicated service-account secret and run:

   Scripts/seal-release-secrets.sh --source-dir intake/release
   Scripts/ci/check-release-config.sh app-store
   Scripts/ci/check-release-config.sh apple-signing
   Scripts/ci/check-release-config.sh gcs
   

7. Treat successful Forgejo release asset publication and successful store upload jobs as separate evidence points.
8. Validate Grafana OpenTofu syntax locally:

   Scripts/grafana-tofu.sh init -backend=false
   Scripts/grafana-tofu.sh validate
   

9. Validate release storage OpenTofu syntax locally:

   Scripts/releases-tofu.sh init -backend=false
   Scripts/releases-tofu.sh validate
   

10. Validate release storage wrapper syntax:

bash -n Scripts/ci/upload-release-storage.sh Scripts/ci/upload-package-storage.sh Scripts/ci/backup-garage-to-gcs.sh Scripts/ci/publish-nix-cache.sh

11. Dispatch a Forgejo workflow without GitHub:

Scripts/forgejo-dispatch.sh build-release.yml --ref main

12. After deploying the forge host, seal:

    BURROW_NIX_CACHE_PUSH_TOKEN from /var/lib/burrow/nix-cache/ci-push-token
    BURROW_GARAGE_BACKUP_ACCESS_KEY_ID from /var/lib/burrow/garage/env
    BURROW_GARAGE_BACKUP_SECRET_ACCESS_KEY from /var/lib/burrow/garage/env
    

Alternatives Considered

• Keep one monolithic release workflow. Rejected because build failures, upload failures, and credential gaps become hard to distinguish.
• Wait for native Windows support before adding a Windows lane. Rejected because packaging and Microsoft Store authentication can be validated with a Rust stub first.
• Add Sparkle to the Apple app in the same change. Rejected because Sparkle changes the client update trust boundary and should be reviewed separately from release plumbing.

Impact on Other Work

• Future Apple release work can fill in signed archive/export details without changing release numbering.
• Future Windows work can replace the stub package with the native client while preserving upload shape.
• Identity and credential follow-up should derive long-lived automation accounts from contributors.nix where they represent Burrow-operated identities.

Decision

Pending.

References

• CONSTITUTION.md
• contributors.nix
• .forgejo/workflows/release-if-needed.yml
• .forgejo/workflows/build-release.yml
• .forgejo/workflows/publish-store-uploads.yml
• Scripts/version.sh
• Scripts/ci/nscloud-cache.sh
• Scripts/ci/ensure-nix.sh
• Scripts/ci/package-apple-artifacts.sh
• Scripts/ci/upload-release-storage.sh
• Scripts/ci/upload-package-storage.sh
• Scripts/ci/publish-nix-cache.sh
• Scripts/ci/backup-garage-to-gcs.sh
• bazel/apple/BUILD.bazel
• Scripts/forgejo-dispatch.sh
• Scripts/forgejo-dispatch-via-host.sh
• Scripts/grafana-tofu.sh
• Scripts/releases-tofu.sh
• .forgejo/workflows/infra-opentofu.yml
• .forgejo/workflows/publish-nix-cache.yml
• .forgejo/workflows/backup-garage-storage.yml
• infra/grafana/
• infra/releases/
• nixos/modules/burrow-garage.nix
• nixos/modules/burrow-observability.nix
• services/grafana/dashboards/burrow-overview.json
• services/grafana/dashboards/headscale.json
• services/grafana/dashboards/observability.json