burrow/docs/OBSERVABILITY.md

Burrow Observability

The forge host imports services.burrow.observability, which wires the first production observability spine:

• Prometheus on 127.0.0.1:9090
• OpenTelemetry collector OTLP gRPC on 127.0.0.1:4317
• OpenTelemetry collector OTLP HTTP on 127.0.0.1:4318
• collector Prometheus export on 127.0.0.1:9464
• Jaeger all-in-one on local ports only
• Grafana at graphs.burrow.net, backed by Authentik SSO

Grafana datasources are provisioned by NixOS:

• prometheus points at local Prometheus
• jaeger points at the local Jaeger query API

OpenTofu owns checked-in dashboards under services/grafana/dashboards/:

• burrow-overview.json
• headscale.json
• observability.json

Metrics

Prometheus scrapes:

• Prometheus self metrics
• node exporter
• systemd exporter
• Grafana /metrics
• Headscale /metrics on 127.0.0.1:9098
• OpenTelemetry collector metrics
• Jaeger admin metrics
• Garage admin metrics when services.burrow.garage.enable = true
• the optional Tailscale exporter when its OAuth environment file is configured

Headscale and Tailscale are treated as the same Tailnet family in dashboards. Headscale exposes a local Prometheus listener through services.burrow.headscale.enableMetrics. Tailscale SaaS metrics require the NixOS Tailscale exporter and an OAuth environment file; the module leaves that disabled until the credential is provisioned.

Traces

Burrow services that emit OpenTelemetry traces should use:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=burrow

The collector exports traces to Jaeger. Jaeger is intentionally local-only for now; Grafana can query it server-side through the provisioned datasource.

Headscale and Tailscale currently enter the spine through Prometheus metrics and systemd state. If an upstream Tailnet component starts emitting OTLP, point it at the same local collector endpoint rather than exposing Jaeger or Prometheus directly.