burrow/evolution/proposals/BEP-0010-platform-services-core-and-distribution.md

BEP-0010 - Platform Services, Core, and Distribution

Status: Draft
Proposal: BEP-0010
Authors: gpt-5.5
Coordinator: gpt-5.5
Reviewers: Pending
Constitution Sections: II, III, IV, V
Implementation PRs: Pending
Decision Date: Pending

Summary

Burrow should grow from a forge-hosted VPN project into a coherent platform: a KMS-backed identity and signing plane, OpenBao-backed secret distribution, private meeting and mail services, an extracted MCP hub for operator and agent workflows, and app builds for Apple, Linux, Android, and later store channels.

The first implementation is deliberately scaffolding-heavy. It adds OpenTofu boundaries for Google KMS, Workload Identity Federation, GCS release backup storage, and OpenBao; enables Garage-backed object storage and a Burrow-owned Nix cache at nix.burrow.net; adds NixOS wiring for Grafana, Prometheus, OpenTelemetry, and Jaeger; adds disabled-by-default NixOS service switch points for vault.burrow.net and meet.burrow.net; introduces a small cross-platform Rust core and Android Kotlin stub; and records the distribution model before stores or public production services are enabled.

Motivation

• The forge now controls source and release automation, but release signing, SAML signing, and runner cloud access still need a stronger non-exportable key story.
• Secrets are still primarily age-managed files. That remains a good bootstrap path, but long-running services and agents need short-lived, attributable runtime access.
• Burrow needs mobile and desktop clients that share protocol logic instead of repeatedly embedding platform-local behavior.
• The GTK Flatpak can be a useful desktop shell, but the actual VPN data plane cannot be assumed to work inside an unprivileged Flatpak sandbox.
• App store, F-Droid, Flatpak, and Play Store distribution should be designed as separate release surfaces with explicit signing, policy, and metadata gates.

Detailed Design

• Add infra/identity as the OpenTofu boundary for Google Cloud identity keys and Authentik-backed Workload Identity Federation:
  • Google project id: project-88c23ce9-918a-470a-b33.
  • Google project number: 416198671487.
  • KMS key ring: burrow-identity.
  • non-exportable signing keys for Authentik signing, SAML CA signing, and release signing.
  • a Forgejo runner service account that can be assumed only through the configured Authentik WIF provider and IAM bindings. Machine-to-machine runner tokens are scoped by the Authentik WIF client ID google-wif.burrow.net, with burrow-automation kept as the human/group policy boundary.
• Add infra/releases as the OpenTofu boundary for Google Cloud Storage-backed release backups:
  • burrow-net-releases backs up build artifacts and Sparkle public channels.
  • burrow-net-packages backs up signed APT, RPM, pacman, Flatpak, and Arch repository trees.
  • burrow-net-nix-cache backs up the Garage attic bucket and remains private by default.
  • Garage is the required S3-compatible primary target for release and package uploads.
  • services.burrow.garage is enabled by the forge host and bootstraps the single-node layout, the attic, burrow-releases, and burrow-packages buckets, scoped owner/write S3 access keys, and a read-only backup key from a host-local environment file.
  • Forgejo jobs authenticate with Authentik-backed WIF before using gcloud storage for backup and Google KMS for signing; rclone and Google service-account JSON keys are not part of the release path.
• Add services.burrow.nixCache for nix.burrow.net:
  • Attic serves https://nix.burrow.net/burrow as the public Burrow cache.
  • Attic stores chunks in the Garage attic bucket through the local S3 API.
  • Workers use the cache only after BURROW_NIX_CACHE_PUBLIC_KEY is published in Forgejo variables.
  • Push access uses /var/lib/burrow/nix-cache/ci-push-token, sealed into Forgejo/OpenBao as BURROW_NIX_CACHE_PUSH_TOKEN, instead of the host-local bootstrap admin token.
  • .forgejo/workflows/publish-nix-cache.yml builds selected flake outputs and publishes their closures to Attic.
• Add services.burrow.observability as the forge host observability spine:
  • Prometheus scrapes local system, Grafana, Headscale, OpenTelemetry collector, and Jaeger metrics.
  • Headscale exposes local metrics on 127.0.0.1:9098.
  • Tailscale SaaS metrics can be enabled later through the Prometheus Tailscale exporter once the OAuth environment file is provisioned.
  • Burrow backend/frontend processes emit OTLP to the local collector at 127.0.0.1:4317 or 127.0.0.1:4318.
  • The collector exports traces to local Jaeger, while Grafana queries Jaeger server-side through a provisioned datasource.
• Add infra/openbao as the OpenTofu boundary for OpenBao:
  • Google KMS seal wrapping.
  • an age/ KV v2 mount for migrated runtime secrets.
  • Authentik OIDC roles for administrators and automation.
  • AppRole policy scaffolding for machine consumers.
• Add NixOS modules under services.burrow.openbao and services.burrow.jitsi:
  • both are imported by the forge host but disabled by default.
  • DNS, reverse proxy, KMS seal, storage, and UDP/video bridge requirements are visible in repo-owned options before activation.
  • live enablement requires a separate deploy with secrets, DNS, and rollback evidence.
• Keep webmail as a second mail phase:
  • inbox.burrow.net is the intended webmail surface.
  • Stalwart is the planned mail server, but the current Nixpkgs module set in this flake does not expose the expected services.stalwart-mail option.
  • Forward Email remains the current production mail path until a Stalwart BEP revision names the exact NixOS module, ports, spam policy, backup target, DKIM rotation, and migration plan.
• Extract an MCP hub to a dedicated compatible.systems repository:
  • target repository: compatible.systems/burrow/mcp-hub.
  • this repository keeps services/mcp-hub/ as the extraction plan and bootstrap manifest until the external repo exists.
  • MCP tools should cover OpenBao, Authentik, KMS/WIF, Forgejo runners, Jitsi, mail, release signing, and Burrow agent identity.
• Create a burrow-core Rust crate as the cross-platform protocol/application core boundary:
  • no direct TUN, NetworkExtension, VpnService, GTK, SwiftUI, or desktop UI dependencies.
  • platform shells link it through a thin FFI crate or native Rust API.
  • target platforms include Linux x86_64/aarch64/riscv64, Android, macOS, iOS, visionOS, and future Windows.
• Add an Android stub:
  • Kotlin app shell in Android/app.
  • Rust JNI/FFI crate in crates/burrow-mobile-ffi.
  • the stub loads the Rust core and displays its version.
  • full VPN support must use Android VpnService; it must not bypass platform disclosure and consent flows.
• Keep Bazel as the stable build orchestration entrypoint:
  • platform lanes expose wrapper targets under bazel/.
  • Bazel actions may call repo-owned scripts, Cargo, Gradle, Xcode, Meson, and OpenTofu while the project migrates deeper native rules selectively.
  • Namespace cache setup remains the fast path for remote Bazel and local disk caches.
• For Linux distribution, split the GUI package from the privileged daemon:
  • DEB, RPM, pacman, AUR, and the NixOS flake/module install burrowd, systemd units, D-Bus policy, and polkit policy.
  • APT, RPM, and pacman repository metadata is generated from release package artifacts and signed through Google KMS-backed OpenPGP signatures.
  • Repository trees publish through the storage wrapper: Garage first when configured, then GCS backup after the runner obtains a short-lived Google credential from Authentik-backed WIF.
  • AUR is a source-build git repository, with burrow-git tracked in this repo before publication to aur.archlinux.org.
  • Flatpak and AppImage packages are GUI-first shells that connect to the host daemon/helper.
  • Flatpak may request background/autostart permission for the GUI, but it must not be responsible for the privileged VPN data plane.
  • PackageKit may be used only as an optional native-package bootstrap on supported distros, with narrow D-Bus permission and polkit authorization.

Security and Operational Considerations

• Google KMS keys are non-exportable and have prevent_destroy enabled.
• Workload Identity Federation is scoped by issuer, audience, mapped attributes, and service-account IAM bindings. Runners should not receive static Google credentials.
• OpenBao bootstrap tokens and AppRole SecretIDs must never enter OpenTofu state. OpenTofu manages mounts, policies, roles, and cloud KMS objects only.
• OpenBao should start behind vault.burrow.net only after:
  • KMS seal key exists and decrypt permission is limited to the service identity.
  • root/admin token bootstrap is recorded out of band.
  • backup/restore and seal/unseal drills are documented.
• Jitsi should start behind meet.burrow.net only after:
  • HTTPS and XMPP domains resolve.
  • UDP media ports are intentionally opened.
  • Authentik guest/member policy is decided.
• Flatpak cannot be the privileged VPN backend by itself. A Flatpak GUI may talk to a host daemon or privileged helper, but opening TUN devices, installing routes, and owning system DNS/routing policy belong outside the sandbox.
• Host service bootstrap should be auditable. A Flatpak may discover the daemon, open native install instructions, or request a host helper through a narrow D-Bus name. It must not require broad system-bus access, host filesystem access, or flatpak-spawn --host as the normal production path.
• Mobile VPN support uses platform VPN APIs:
  • Android: VpnService with Play policy disclosure for Play Store builds.
  • iOS and visionOS: NetworkExtension packet tunnel provider.
• Release signing, SAML CA signing, and Authentik signing should remain separate keys even when they share a KMS key ring.
• Package repository signing should stay split by repository format. APT, RPM, pacman, Flatpak, and AUR source surfaces each get a dedicated KMS key so a compromised publisher path does not automatically sign every Linux channel.
• Observability ingress is local-first. Prometheus, the OpenTelemetry collector, and Jaeger bind to loopback; Grafana is the authenticated public surface.
• Jaeger should not be exposed publicly until an Authentik-protected route or equivalent access boundary is added.
• GCS is a backup target, not the storage abstraction. Future multi-cloud failover should add explicit provider mirrors or S3-compatible endpoints behind the storage wrappers.
• Garage needs real host storage for object data and metadata. It replicates across Garage nodes, but it does not manage multiple external storage backends. GCS and later providers should be explicit mirror/backup jobs.
• .forgejo/workflows/backup-garage-storage.yml is the first explicit mirror. It uses a read-only Garage key and Authentik-backed WIF to copy Garage bucket contents to GCS without rclone or static Google service-account keys.

Contributor Playbook

1. Validate BEP metadata:

   python3 Scripts/check-bep-metadata.py
   

2. Check the Android/Rust stub:

   cargo check --locked -p burrow-mobile-ffi
   bazel build //bazel/android:check_stub_stamp
   

3. Validate identity OpenTofu syntax:

   Scripts/identity-tofu.sh init -backend=false
   Scripts/identity-tofu.sh validate
   

4. Validate release storage OpenTofu syntax:

   Scripts/releases-tofu.sh init -backend=false
   Scripts/releases-tofu.sh validate
   

5. Validate observability Nix wiring and dashboard JSON:

   nix eval .#nixosConfigurations.burrow-forge.config.services.prometheus.scrapeConfigs
   jq empty services/grafana/dashboards/headscale.json services/grafana/dashboards/observability.json
   

6. Validate the Nix cache module wiring:

   nix eval .#nixosConfigurations.burrow-forge.config.services.atticd.settings.storage --json
   

7. Validate cache publishing and Garage backup wrapper syntax:

   bash -n Scripts/ci/publish-nix-cache.sh Scripts/ci/backup-garage-to-gcs.sh
   

8. Validate OpenBao OpenTofu syntax:

   Scripts/openbao-tofu.sh init -backend=false
   Scripts/openbao-tofu.sh validate
   

9. Before enabling vault.burrow.net, create or import the Google KMS seal key, configure OpenBao bootstrap access, and record a restore test.
10. Before enabling meet.burrow.net, verify DNS, TLS, UDP reachability, and Authentik access policy.
11. Before distributing Flatpak as more than a GUI, prove the host daemon/helper path on a real Linux desktop.
12. Before Play Store distribution, complete the VpnService declaration and data-safety review.
13. Before relying on PackageKit bootstrap, test Fedora, Debian/Ubuntu, and an immutable/atomic desktop separately. PackageKit support is not universal and should degrade to native install instructions.
14. Before promoting package repositories to stable, run the repository publisher, import the generated public key on a clean VM for each package family, and install Burrow from that repository rather than from the loose artifact.

Alternatives Considered

• Enable OpenBao and Jitsi immediately on the forge host. Rejected because both change public ingress and security boundaries before secrets, DNS, and rollback evidence are complete.
• Store Google service-account JSON in Forgejo secrets. Rejected because WIF gives the runner an attributable short-lived credential path without static cloud keys.
• Put Android directly on the existing daemon crate. Rejected for now because the daemon carries OS-specific TUN and service assumptions. A small cross-platform core gives mobile a safer first boundary.
• Treat Flatpak as the complete Linux VPN package. Rejected because routing, TUN, and DNS changes need host authority.
• Use broad Flatpak host escape permissions to fork the daemon directly. Rejected because that defeats the value of Flatpak and makes review harder than shipping a small native daemon package with explicit polkit policy.
• Use one package signing key for every Linux repository. Rejected because format-specific keys give cleaner revocation and narrower IAM grants.

Impact on Other Work

• BEP-0009 release work gains identity and app-distribution follow-up lanes.
• BEP-0005 and BEP-0006 remain authoritative for daemon/client boundaries.
• BEP-0004 remains authoritative for current hosted mail until a Stalwart revision supersedes it.
• MCP extraction should not move forge runner code until the external compatible.systems repository exists and import paths are stable.

Decision

Pending.

References

• CONSTITUTION.md
• contributors.nix
• evolution/proposals/BEP-0004-hosted-mail-and-saas-identity.md
• evolution/proposals/BEP-0005-daemon-ipc-and-apple-boundary.md
• evolution/proposals/BEP-0006-tailnet-authority-first-control-plane.md
• evolution/proposals/BEP-0009-release-infrastructure-and-store-upload-pipeline.md
• infra/identity/
• infra/releases/
• infra/openbao/
• docs/OBSERVABILITY.md
• docs/NIX_CACHE.md
• .forgejo/workflows/publish-nix-cache.yml
• .forgejo/workflows/backup-garage-storage.yml
• Scripts/ci/publish-nix-cache.sh
• Scripts/ci/backup-garage-to-gcs.sh
• package/repositories/
• nixos/modules/burrow-garage.nix
• nixos/modules/burrow-nix-cache.nix
• nixos/modules/burrow-observability.nix
• nixos/modules/burrow-openbao.nix
• nixos/modules/burrow-jitsi.nix
• crates/burrow-core/
• crates/burrow-mobile-ffi/
• Android/
• bazel/android/
• services/mcp-hub/