# Burrow Observability

The forge host imports `services.burrow.observability`, which wires the first
production observability spine:

- Prometheus on `127.0.0.1:9090`
- OpenTelemetry collector OTLP gRPC on `127.0.0.1:4317`
- OpenTelemetry collector OTLP HTTP on `127.0.0.1:4318`
- collector Prometheus export on `127.0.0.1:9464`
- Jaeger all-in-one on local ports only
- Grafana at `graphs.burrow.net`, backed by Authentik SSO

Grafana datasources are provisioned by NixOS:

- `prometheus` points at local Prometheus
- `jaeger` points at the local Jaeger query API

OpenTofu owns checked-in dashboards under `services/grafana/dashboards/`:

- `burrow-overview.json`
- `headscale.json`
- `observability.json`

## Metrics

Prometheus scrapes:

- Prometheus self metrics
- node exporter
- systemd exporter
- Grafana `/metrics`
- Headscale `/metrics` on `127.0.0.1:9098`
- OpenTelemetry collector metrics
- Jaeger admin metrics
- Garage admin metrics when `services.burrow.garage.enable = true`
- the optional Tailscale exporter when its OAuth environment file is configured

Headscale and Tailscale are treated as the same Tailnet family in dashboards.
Headscale exposes a local Prometheus listener through
`services.burrow.headscale.enableMetrics`. Tailscale SaaS metrics require the
NixOS Tailscale exporter and an OAuth environment file; the module leaves that
disabled until the credential is provisioned.

## Traces

Burrow services that emit OpenTelemetry traces should use:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=burrow
```

The collector exports traces to Jaeger. Jaeger is intentionally local-only for
now; Grafana can query it server-side through the provisioned datasource.

Headscale and Tailscale currently enter the spine through Prometheus metrics and
systemd state. If an upstream Tailnet component starts emitting OTLP, point it
at the same local collector endpoint rather than exposing Jaeger or Prometheus
directly.