Wire Forge-native release infrastructure
Some checks failed
Cache: Publish Nix / Publish Nix Cache (push) Waiting to run
Build Rust / Cargo Test (push) Successful in 4m14s
Build Site / Next.js Build (push) Failing after 6s
Infra: OpenTofu / OpenTofu (grafana) (push) Successful in 5s
Lint Governance / BEP Metadata (push) Successful in 0s
Build: Android / Android Rust Core Stub (push) Failing after 23s

This commit is contained in:
Conrad Kramer 2026-06-07 05:51:12 -07:00
parent 97c569fb35
commit 002bd382e9
199 changed files with 14268 additions and 185 deletions

60
docs/OBSERVABILITY.md Normal file
View file

@ -0,0 +1,60 @@
# Burrow Observability
The forge host imports `services.burrow.observability`, which wires the first
production observability spine:
- Prometheus on `127.0.0.1:9090`
- OpenTelemetry collector OTLP gRPC on `127.0.0.1:4317`
- OpenTelemetry collector OTLP HTTP on `127.0.0.1:4318`
- collector Prometheus export on `127.0.0.1:9464`
- Jaeger all-in-one on local ports only
- Grafana at `graphs.burrow.net`, backed by Authentik SSO
Grafana datasources are provisioned by NixOS:
- `prometheus` points at local Prometheus
- `jaeger` points at the local Jaeger query API
OpenTofu owns checked-in dashboards under `services/grafana/dashboards/`:
- `burrow-overview.json`
- `headscale.json`
- `observability.json`
## Metrics
Prometheus scrapes:
- Prometheus self metrics
- node exporter
- systemd exporter
- Grafana `/metrics`
- Headscale `/metrics` on `127.0.0.1:9098`
- OpenTelemetry collector metrics
- Jaeger admin metrics
- Garage admin metrics when `services.burrow.garage.enable = true`
- the optional Tailscale exporter when its OAuth environment file is configured
Headscale and Tailscale are treated as the same Tailnet family in dashboards.
Headscale exposes a local Prometheus listener through
`services.burrow.headscale.enableMetrics`. Tailscale SaaS metrics require the
NixOS Tailscale exporter and an OAuth environment file; the module leaves that
disabled until the credential is provisioned.
## Traces
Burrow services that emit OpenTelemetry traces should use:
```sh
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=burrow
```
The collector exports traces to Jaeger. Jaeger is intentionally local-only for
now; Grafana can query it server-side through the provisioned datasource.
Headscale and Tailscale currently enter the spine through Prometheus metrics and
systemd state. If an upstream Tailnet component starts emitting OTLP, point it
at the same local collector endpoint rather than exposing Jaeger or Prometheus
directly.