burrow/evolution/proposals/BEP-0009-release-infrastructure-and-store-upload-pipeline.md

BEP-0009 - Release Infrastructure and Store Upload Pipeline

Status: Draft
Proposal: BEP-0009
Authors: gpt-5.5
Coordinator: gpt-5.5
Reviewers: Pending
Constitution Sections: II, III, IV, V
Implementation PRs: Pending
Decision Date: Pending

Summary

Burrow needs a release pipeline that behaves like the project infrastructure it is replacing: release eligibility is detected from forge build tags, build artifacts are produced per platform, upload lanes are separate from build lanes, and store credentials are explicit automation inputs rather than hidden local state.

The first implementation creates the release spine now: release-if-needed dispatches a build when HEAD has not been tagged as released, build-release packages current artifacts on Namespace-backed runners, and publish-store-uploads performs guarded upload steps for App Store Connect, Sparkle, Microsoft Store, and storage-backed public channels after Authentik-backed Google WIF authentication.

The same change also makes the forge itself the release authority: release tags prefer the Burrow Forgejo remote, dispatch helpers talk to Forgejo directly with a live-host fallback, runner identity is discoverable from persistent host state, repository mirror sync is disabled for the canonical Burrow repository, nix.burrow.net provides a Burrow-owned Nix cache for workers, and Grafana observability is bootstrapped with Nix-owned Prometheus/OpenTelemetry/Jaeger service state plus OpenTofu-owned dashboards.

Motivation

• Burrow already controls its forge and runner infrastructure, but the release path only produced a Linux CLI tarball.
• Apple, Linux, Sparkle, and future Windows outputs need the same build-number and artifact layout so release evidence is inspectable.
• The Namespace account should be part of the live runner surface, including macOS and Windows lanes, instead of sitting idle behind unused configuration.
• Authentication and store credentials should be named and checked at lane boundaries so dedicated accounts can be added without rewriting workflows.

Detailed Design

• Use lightweight builds/<number> tags as the authoritative release watermark.
• Keep Scripts/version.sh status as the release-needed check:
  • clean means HEAD is exactly the latest authoritative build tag.
  • dirty means the release workflow should create a new build number.
• Split workflows:
  • .forgejo/workflows/release-if-needed.yml checks release status and dispatches build-release.yml.
  • .forgejo/workflows/build-release.yml builds and stages artifacts.
  • .forgejo/workflows/publish-store-uploads.yml uploads staged artifacts to external stores and public channels.
• Stage artifacts under dist/builds/<build-number>/<platform>/.
• Publish generic release assets to a Forgejo release named Build <number>.
• Upload release artifacts through Scripts/ci/upload-release-storage.sh. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the burrow-net-releases GCS backup bucket unless BURROW_RELEASE_GCS_BACKUP=false.
• Publish signed package repository trees through Scripts/ci/upload-package-storage.sh. The wrapper requires Garage as the primary S3-compatible target by default, then mirrors to the burrow-net-packages GCS backup bucket unless BURROW_PACKAGE_GCS_BACKUP=false.
• Publish selected Nix closures through .forgejo/workflows/publish-nix-cache.yml and Scripts/ci/publish-nix-cache.sh. The workflow logs into nix.burrow.net with the scoped BURROW_NIX_CACHE_PUSH_TOKEN generated by host bootstrap and skips when the token is absent unless explicitly required.
• Mirror Garage buckets through .forgejo/workflows/backup-garage-storage.yml and Scripts/ci/backup-garage-to-gcs.sh. The scheduled workflow uses the host-generated Garage read-only backup key, Authentik-backed Google WIF, and gcloud storage rsync to back up burrow-releases, burrow-packages, and attic to GCS.
• Build Apple artifacts on namespace-profile-macos-large through Bazel wrapper targets:
  • //bazel/apple:release_ios_stamp
  • //bazel/apple:release_macos_stamp These targets wrap the existing Xcode project and give the workflow one stable entrypoint that shares the same Bazel/Nix cache setup while Burrow decides whether deeper rules_apple targets are worth the migration.
• Use Namespace cache setup before Bazel/Nix work. Prefer the Namespace Bazel remote cache when nsc is authenticated and fall back to a local disk/repository cache when remote setup is unavailable. Cache setup is optional release acceleration: Bazel-only cache steps must not install filesystem-cache helpers such as spacectl, and Namespace cache probes must be timeout-bounded so Apple builds can continue without the remote cache. Apple workflows should call the fail-open cache helper directly instead of entering the full Nix development shell just to decide whether a cache is available. macOS runners should use the local Bazel disk/repository cache path and skip the nsc remote-cache auth probe, which is optional and can block without GNU timeout support.
• The Nix CI shell must not pin Apple builds to a Bazel major that disagrees with .bazelversion. Prefer Bazelisk or a Bazel 9 package so the release lane uses the repository-declared Bazel 9.1.0 toolchain.
• Moving tarball URLs are not acceptable for release-critical Nix inputs. The forge worker lockfile must resolve nixpkgs through a fixed Git revision so fresh workers do not fail when a branch tarball no longer matches a stored NAR hash.
• The CI shell should prefer currently supported Node runtimes when nixpkgs marks an older major insecure. Release workers must fail on product build errors, not on avoidable toolchain end-of-life policy checks.
• Bazel repository and disk caches must live outside the checked-out repository on macOS runners. Bazel 9 rejects a repository contents cache inside the main repo because it can create spurious build graph failures.
• Bazel Apple genrules must explicitly pass the Nix CI shell tool path and PROTOC into the action environment. Xcode build phases run with a reduced path, and the Rust Network Extension build depends on protoc.
• Forgejo release workflows must use the artifact action generation supported by the deployed forge. The v4 artifact actions assume GitHub's newer artifact service and fail on this Forgejo install before downstream signing/upload jobs can consume the staged Apple artifacts.
• The release secret decrypt action must be self-contained after the Nix bootstrap step. Linux Namespace signing jobs install Nix at runtime, so the bootstrap writes Nix profile directories to GITHUB_PATH, the decrypt action resolves Nix from those profile paths, and the action builds .#age before falling back to .#agenix. The decrypt step must not require Python before the Nix CI shell is active.
• Authentik-backed Google WIF must exchange tokens against Authentik's OAuth token endpoint, not a provider-slug child path. The helper should prefer an explicit BURROW_AUTHENTIK_WIF_TOKEN_ENDPOINT, then OIDC discovery, then the Authentik /application/o/token/ fallback derived from the configured issuer.
• The Google WIF Authentik application is a machine-to-machine client-credentials provider. It must not default to a human group policy binding, because the token exchange has no interactive user and Authentik will reject the grant before Google WIF can validate the issued JWT.
• Forgejo should store the Authentik WIF credential as the client-credentials client_secret form value. For service-account authentication, that value is base64("<service-account-username>:<app-password-token>"); the helper must send it as form data, not HTTP Basic auth.
• Produce unsigned Apple validation artifacts when signing is absent, but require signing for an App Store requested iOS release and for a Sparkle tester release. Uploadable artifacts are produced only after provisioning profiles can be synced from App Store Connect credentials and the relevant Apple certificate is proven to match the selected Google KMS key.
• Resolve release credentials through age/agenix before signed Apple lanes run. Forgejo secrets should carry only the runner age identity fallback and the Authentik WIF client secret until OpenBao can mint the runner token directly. App Store Connect keys and signing certificates live as sealed files under secrets/.
• Persist the forge runner SSH key as /var/lib/forgejo-runner-agent/age_keystore and export BURROW_RUNNER_AGE_IDENTITY_PATH so jobs can resolve agenix identities without embedding long-lived material in every workflow.
• Add Scripts/forgejo-dispatch.sh and Scripts/forgejo-dispatch-via-host.sh:
  • local dispatch uses an explicit token, local intake token, or the sealed forgejo-nsc dispatcher config when available;
  • host fallback SSHes to the live forge and reads the agenix-materialized dispatcher config there;
  • neither path relies on a GitHub remote or GitHub workflow API.
• Default build tag reads and pushes to the Burrow Forgejo remote when that remote exists. BURROW_VERSION_REMOTE remains the override for unusual migrations.
• Keep the canonical Forgejo repo as an ordinary source repository, not a mirror. The host config disables new mirror creation and removes mirror/push-mirror rows for the canonical Burrow repository at activation time.
• Add .forgejo/workflows/infra-opentofu.yml and Scripts/grafana-tofu.sh for Grafana API-managed folders and dashboards. NixOS owns the Grafana service, SSO, reverse proxy, datasources, Prometheus, OpenTelemetry collector, Jaeger, secrets, and boot-time configuration; OpenTofu owns API-created dashboard objects.
• Sync provisioning profiles from App Store Connect in CI using the decrypted API key:
  • iOS App Store profiles for net.burrow.app and net.burrow.app.network.
  • macOS Developer ID profiles for the same bundle identifiers. Provisioning profile .age files are not part of the normal release path.
• Add a non-exportable Google KMS RSA-2048/SHA-256 key named apple-developer-id-application in the Burrow identity key ring. The manual apple-developer-id-kms-csr.yml workflow uses Authentik-backed Google WIF to generate a standard Developer ID Application CSR without exporting private key material.
• Treat Developer ID certificate issuance as a Developer website or Xcode step until Apple documents App Store Connect API creation for Developer ID certificates. The returned .cer is public release material; the private key remains in Google KMS and must not be converted into .p12 form.
• The first KMS-backed G2 Developer ID Application certificate is 9JKN6HXBHC, stored at Apple/Certificates/developer-id-application-9JKN6HXBHC.cer, and expires on 2031-06-08.
• Add a non-exportable Google KMS RSA-2048/SHA-256 key named apple-ios-distribution in the Burrow identity key ring. The KMS-backed CSR can be submitted through the App Store Connect API with Scripts/apple/create-asc-certificate.mjs.
• The first KMS-backed iOS Distribution certificate is 3G42677598, stored at Apple/Certificates/ios-distribution-3G42677598.cer, and expires on 2027-06-07.
• When BURROW_IOS_DISTRIBUTION_CERTIFICATE_ID is set, provisioning-profile sync must verify App Store profiles reference that certificate. Stale profiles with the same name may be deleted and recreated so CI does not silently sign against an older distribution identity.
• Apple release signing uses a staged KMS flow because Google publishes the Cloud KMS PKCS#11 library for Linux release runners, not macOS runners:
  • macOS Namespace builders compile unsigned iOS device and macOS app bundles with the existing Xcode project.
  • Linux release runners download those staged artifacts, authenticate to Google through Authentik WIF, verify the Apple .cer public key matches the intended Google KMS key version, embed the synced provisioning profiles, and sign the app/extension bundles with the patched rcodesign PKCS#11 path.
  • The signed iOS IPA is uploaded to App Store Connect, and the signed macOS ZIP is used as the Sparkle tester artifact.
• Add a Google KMS key named sparkle-ed25519 for small Sparkle signing probes and future signing-service work. It uses EC_SIGN_ED25519 with software protection level because Google KMS rejects Ed25519 for HSM protection. Google KMS also rejects normal macOS release archives as direct Ed25519 messages, so the tester pipeline signs full-size Sparkle archives directly from the decrypted SPARKLE_EDDSA_KEY_PATH release seed until a compatible KMS-backed Sparkle signing service is introduced. macOS builds embed public EdDSA key uugZuJeqvvKd91NZ6F1Fv2cQenUbIG/ZW3L9MuaEz30=.
• Use Namespace labels for platform lanes:
  • namespace-profile-linux-medium
  • namespace-profile-macos-large
  • namespace-profile-windows-large
• Use dedicated Apple distribution labels for the focused tester-distribution workflow so stale generic Namespace jobs cannot consume Apple release runners:
  • namespace-profile-linux-medium-apple-v2
  • namespace-profile-macos-large-apple-v2 Label revisions are allowed when earlier accepted workflows have stale queued jobs; the autoscaler should serve only the current focused label set.
• Configure macOS Namespace targets with the Compute API CPU-by-memory shape format. 12x28 is the Burrow large macOS baseline; platform-prefixed values such as macos/arm64:8x16 are not accepted by the macOS launcher.
• Treat macOS shape fallback as part of the release lane rather than as a best-effort debug path. When large shapes are quota-constrained, the dispatcher may try smaller accepted shapes, but every fallback must have a bounded create timeout and additional shapes must reuse the final timeout policy instead of crashing the dispatcher.
• Keep the shared Linux runner work directory out of macOS bootstraps. The dispatcher normalizes macOS runners to /tmp/forgejo-runner when the shared config points at Linux state paths such as /var/lib/forgejo-runner.
• Do not destroy Namespace Linux runners until nsc describe shows a real terminal container state or tombstone. Empty describe payloads mean the environment is still starting, not that the one-job runner has finished.
• Keep the autoscaler-to-dispatcher HTTP timeout longer than the runner TTL. Dispatch requests intentionally block until the one-job runner exits, so short HTTP client timeouts cancel macOS provisioning mid-bootstrap.
• Run forge Namespace services with the repo-owned nsc package and keep it current with the token format expected by Namespace. nsc 0.0.520 is the first required baseline for revokable token-file auth. The dispatcher and autoscaler systemd units must also use the repo-built packages so runner fixes ship with Burrow deploys instead of staying pinned to the module input.
• Render the sealed Namespace runner token with nsc token create --token_file instead of nsc auth generate-dev-token. The dev-token output is for direct API/registry access, while the dispatcher needs the JSON token-file format consumed by nsc run.
• Add a Rust burrow-windows-stub binary so the Windows lane can validate packaging and upload mechanics before native Windows networking support is implemented.
• Treat Sparkle as a macOS public-channel upload lane:
  • Generate staged appcast content from the signed macOS artifact when present.
  • Publish sparkle/<channel>/, sparkle/appcast.xml, and sparkle/default/ pointers from publish-store-uploads.
  • The macOS app embeds Sparkle and consumes the https://releases.burrow.net/sparkle/appcast.xml feed.
• Treat TestFlight distribution as a retryable post-upload step. Apple build processing can exceed 30 minutes after a successful IPA upload, so apple-distribute-testers and publish-store-uploads wait up to 7200 seconds by default and allow Forgejo variable TESTFLIGHT_WAIT_PROCESSING_TIMEOUT_SECONDS to tune that ceiling without a repo change. TestFlight wait jobs must use namespace-profile-linux-testflight or namespace-profile-macos-large-testflight; the generic Linux and Apple macOS profiles expire before the 7200-second processing window. The autoscaler-to-dispatcher timeout must stay above those runner TTLs because dispatch requests block until the one-job runner exits. publish-store-uploads has a separate no-needs distribute-ios-testflight retry job for distribute_testflight=true and upload_app_store=false, so skipped upload/signing jobs cannot suppress distribution of an already-uploaded build. distribute-testflight is the narrower one-job retry workflow when only App Store Connect processing and tester assignment need to be retried.
• Internal TestFlight distribution must not submit the build for external beta review. The internal lane waits for processing, sets export-compliance metadata, uses Fastlane skip_submission:true, and relies on App Store Connect's configured internal tester group availability. External TestFlight distribution requires explicit groups plus beta-review metadata before it can be enabled.
• Add infra/releases as the OpenTofu boundary for the GCS release and package repository backup buckets. Garage is the required S3-compatible primary distribution target after host bootstrap. infra/identity remains the boundary for Google KMS signing keys, the Authentik WIF pool/provider, and the runner service account.
• Serve releases.burrow.net and packages.burrow.net from the forge Caddy edge as stable Burrow domains backed by the GCS release and package buckets. Appcast enclosure URLs and package repository metadata must use those Burrow domains instead of provider-specific object-storage URLs.

Security and Operational Considerations

• Store credentials are consumed only by signed build or upload jobs, not by ordinary build/test jobs.
• GCS backup and Google KMS access must use Authentik-backed Workload Identity Federation against project project-88c23ce9-918a-470a-b33; workflows must not use Google service-account JSON keys.
• Garage uploads use S3-compatible access keys scoped to release/package buckets. They are required by default; GCS is a backup mirror, not the success fallback.
• Garage backup uses a separate read-only key spanning the release, package, and Attic buckets. That key may be sealed into Forgejo/OpenBao for backup jobs, but it must not be reused for release/package writes.
• Attic cache publishing uses a scoped CI push token for the burrow cache instead of the bootstrap admin token.
• rclone is not part of the release path. Multi-cloud failover should be added through provider-specific or S3-compatible wrappers with explicit credentials and evidence.
• Required upload credentials are checked explicitly after ./.forgejo/actions/decrypt-release-secrets materializes them from age:
  • secrets/apple/appstore-connect.env.age: ASC_API_KEY_ID, ASC_API_ISSUER_ID, and base64-encoded .p8 key material.
  • KMS-backed Apple signing must not use .p12 private-key material. The legacy secrets/apple/distribution-signing.env.age path remains accepted only for emergency keychain signing and should not be used for the normal tester release lane.
  • Sparkle appcasts are signed with Google KMS by default; the legacy secrets/apple/sparkle.env.age path remains accepted only for emergency local EdDSA signing.
  • future Microsoft Partner Center credentials
• The temporary CI keychain password is always the distribution certificate password; there is no separate keychain-password release secret.
• KMS-backed Apple certificates do not satisfy the existing Xcode/keychain import path. Shipping with these certificates requires a signer that can delegate Apple code-signing operations to Google KMS.
• iOS KMS signing must filter provisioning-profile entitlements through the checked-in iOS entitlement templates before signing. App Store Connect validates the bundle signature, not just the profile, and rejects profile-only capabilities such as system extensions or unsupported NetworkExtension values when they leak into an iOS app or extension signature.
• The App target must keep Assets.xcassets in its Resources phase. Xcode derives the CFBundleIcons metadata and standalone App Store icon PNGs from that resource membership during iOS builds.
• App Store Connect upload steps must fail on altool validation output as well as non-zero process exits. Apple can emit a textual upload failure before the workflow reaches TestFlight processing, and downstream tester waits must not run unless the IPA upload was actually accepted.
• If the first key ring or Developer ID key is bootstrapped before the OpenTofu remote state backend is available locally, the live resources must be imported into infra/identity before enabling managed identity applies.
• Dedicated store accounts should be sealed with agenix before enabling always-on uploads. The runner only needs an age identity such as /var/lib/forgejo-runner-agent/age_keystore; AGE_FORGE_SSH_KEY is a fallback bootstrap secret, not a place for release material.
• The canonical identity registry reserves release@burrow.net for release automation and namespace@burrow.net for Namespace runner capacity credentials; secrets stay outside contributors.nix.
• Grafana starts with a sealed admin password and a pending OIDC client secret. Local admin login remains available until the real OIDC secret is rotated and the Authentik reconciliation service creates the Grafana provider.
• OpenTofu apply/import modes require an explicit confirmation and remote state. Validation may run without remote state.
• Build tags are pushed only after artifacts have been assembled for the build number.
• External TestFlight distribution and public app-store rollout must stay explicit; internal or private lanes can be defaulted only after credentials and metadata are verified.

Contributor Playbook

1. Check whether a release is needed:

   Scripts/version.sh status
   

2. Build local artifacts:

   BUILD_NUMBER=local Scripts/ci/build-release-artifacts.sh all
   

3. Build Apple release validation artifacts on macOS:

   BUILD_NUMBER=local bazel build //bazel/apple:release_ios_stamp //bazel/apple:release_macos_stamp
   

4. Verify the Windows stub still builds:

   cargo test --locked -p burrow-windows-stub
   

5. Validate BEP metadata:

   python3 Scripts/check-bep-metadata.py
   

6. Before turning on an upload lane, seal the dedicated service-account secret and run:

   Scripts/seal-release-secrets.sh --source-dir intake/release
   Scripts/ci/check-release-config.sh app-store
   Scripts/ci/check-release-config.sh apple-signing
   Scripts/ci/check-release-config.sh gcs
   

7. Treat successful Forgejo release asset publication and successful store upload jobs as separate evidence points.
8. Validate Grafana OpenTofu syntax locally:

   Scripts/grafana-tofu.sh init -backend=false
   Scripts/grafana-tofu.sh validate
   

9. Validate release storage OpenTofu syntax locally:

   Scripts/releases-tofu.sh init -backend=false
   Scripts/releases-tofu.sh validate
   

10. Validate release storage wrapper syntax:

bash -n Scripts/ci/upload-release-storage.sh Scripts/ci/upload-package-storage.sh Scripts/ci/backup-garage-to-gcs.sh Scripts/ci/publish-nix-cache.sh

11. Dispatch a Forgejo workflow without GitHub:

Scripts/forgejo-dispatch.sh build-release.yml --ref main

12. After deploying the forge host, seal:

    BURROW_NIX_CACHE_PUSH_TOKEN from /var/lib/burrow/nix-cache/ci-push-token
    BURROW_GARAGE_BACKUP_ACCESS_KEY_ID from /var/lib/burrow/garage/env
    BURROW_GARAGE_BACKUP_SECRET_ACCESS_KEY from /var/lib/burrow/garage/env
    

Alternatives Considered

• Keep one monolithic release workflow. Rejected because build failures, upload failures, and credential gaps become hard to distinguish.
• Wait for native Windows support before adding a Windows lane. Rejected because packaging and Microsoft Store authentication can be validated with a Rust stub first.
• Add Sparkle to the Apple app in the same change. Rejected because Sparkle changes the client update trust boundary and should be reviewed separately from release plumbing.

Impact on Other Work

• Future Apple release work can fill in signed archive/export details without changing release numbering.
• Future Windows work can replace the stub package with the native client while preserving upload shape.
• Identity and credential follow-up should derive long-lived automation accounts from contributors.nix where they represent Burrow-operated identities.

Decision

Pending.

References

• CONSTITUTION.md
• contributors.nix
• .forgejo/workflows/release-if-needed.yml
• .forgejo/workflows/build-release.yml
• .forgejo/workflows/publish-store-uploads.yml
• Scripts/version.sh
• Scripts/ci/nscloud-cache.sh
• Scripts/ci/ensure-nix.sh
• Scripts/ci/package-apple-artifacts.sh
• Scripts/ci/upload-release-storage.sh
• Scripts/ci/upload-package-storage.sh
• Scripts/ci/publish-nix-cache.sh
• Scripts/ci/backup-garage-to-gcs.sh
• bazel/apple/BUILD.bazel
• Scripts/forgejo-dispatch.sh
• Scripts/forgejo-dispatch-via-host.sh
• Scripts/grafana-tofu.sh
• Scripts/releases-tofu.sh
• .forgejo/workflows/infra-opentofu.yml
• .forgejo/workflows/publish-nix-cache.yml
• .forgejo/workflows/backup-garage-storage.yml
• infra/grafana/
• infra/releases/
• nixos/modules/burrow-garage.nix
• nixos/modules/burrow-observability.nix
• services/grafana/dashboards/burrow-overview.json
• services/grafana/dashboards/headscale.json
• services/grafana/dashboards/observability.json