002bd382e9
Some checks failed
Cache: Publish Nix / Publish Nix Cache (push) Waiting to run
Build Rust / Cargo Test (push) Successful in 4m14s
Build Site / Next.js Build (push) Failing after 6s
Infra: OpenTofu / OpenTofu (grafana) (push) Successful in 5s
Lint Governance / BEP Metadata (push) Successful in 0s
Build: Android / Android Rust Core Stub (push) Failing after 23s
60 lines
2 KiB
Markdown
60 lines
2 KiB
Markdown
# Burrow Observability
|
|
|
|
The forge host imports `services.burrow.observability`, which wires the first
|
|
production observability spine:
|
|
|
|
- Prometheus on `127.0.0.1:9090`
|
|
- OpenTelemetry collector OTLP gRPC on `127.0.0.1:4317`
|
|
- OpenTelemetry collector OTLP HTTP on `127.0.0.1:4318`
|
|
- collector Prometheus export on `127.0.0.1:9464`
|
|
- Jaeger all-in-one on local ports only
|
|
- Grafana at `graphs.burrow.net`, backed by Authentik SSO
|
|
|
|
Grafana datasources are provisioned by NixOS:
|
|
|
|
- `prometheus` points at local Prometheus
|
|
- `jaeger` points at the local Jaeger query API
|
|
|
|
OpenTofu owns checked-in dashboards under `services/grafana/dashboards/`:
|
|
|
|
- `burrow-overview.json`
|
|
- `headscale.json`
|
|
- `observability.json`
|
|
|
|
## Metrics
|
|
|
|
Prometheus scrapes:
|
|
|
|
- Prometheus self metrics
|
|
- node exporter
|
|
- systemd exporter
|
|
- Grafana `/metrics`
|
|
- Headscale `/metrics` on `127.0.0.1:9098`
|
|
- OpenTelemetry collector metrics
|
|
- Jaeger admin metrics
|
|
- Garage admin metrics when `services.burrow.garage.enable = true`
|
|
- the optional Tailscale exporter when its OAuth environment file is configured
|
|
|
|
Headscale and Tailscale are treated as the same Tailnet family in dashboards.
|
|
Headscale exposes a local Prometheus listener through
|
|
`services.burrow.headscale.enableMetrics`. Tailscale SaaS metrics require the
|
|
NixOS Tailscale exporter and an OAuth environment file; the module leaves that
|
|
disabled until the credential is provisioned.
|
|
|
|
## Traces
|
|
|
|
Burrow services that emit OpenTelemetry traces should use:
|
|
|
|
```sh
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
|
|
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
|
|
OTEL_SERVICE_NAME=burrow
|
|
```
|
|
|
|
The collector exports traces to Jaeger. Jaeger is intentionally local-only for
|
|
now; Grafana can query it server-side through the provisioned datasource.
|
|
|
|
Headscale and Tailscale currently enter the spine through Prometheus metrics and
|
|
systemd state. If an upstream Tailnet component starts emitting OTLP, point it
|
|
at the same local collector endpoint rather than exposing Jaeger or Prometheus
|
|
directly.
|