11 — Monitoring & Observability
Three signals, one correlation key: every request/session carries an OpenTelemetry trace_id that appears in traces, structured logs, and (as exemplars) metrics — from gateway to SFU webhook.
Architecture
- Prometheus per region scrapes everything; Thanos (sidecar + query) provides the global view and long-term storage in object storage. Grafana queries one global endpoint.
- Logs: structured JSON (
ts, level, service, trace_id, tenant_id, room_id, msg, fields) → OTel Collector → OpenSearch (ILM: hot 7 d, warm 30 d, delete 90 d; auth events 1 y). - Traces: OTel SDK in every service, W3C
traceparentpropagated (REST headers, WS envelope, Kafka headers). Tail-based sampling: 100 % of errors + joins slower than 2 s, 1 % baseline. - Client telemetry is a first-class source: SDK QoS beacons (02 §8) land in ClickHouse — the only place user-experienced quality is truly visible.
Metric Catalog (the ones that matter)
Platform KPIs (business + health)
| Metric | Source |
|---|---|
active_users (gauge, per region/tenant) | Presence |
concurrent_rooms / room_participants histogram | Room/Presence |
join_latency_seconds (click→media_ready, histogram) | Signaling + SFU webhook |
join_success_ratio | Room + Signaling |
session_duration_seconds | Room sessions |
Media plane
| Metric | Source |
|---|---|
sfu_bandwidth_bytes{direction} per node | LiveKit |
sfu_participants / sfu_tracks per node | LiveKit |
sfu_packet_loss_pct, sfu_rtt_ms, sfu_jitter_ms (per-node aggregates) | LiveKit |
turn_allocations_active, turn_relayed_bytes per node/cluster | Coturn exporter |
turn_relay_ratio (sessions on relay / total) | TURN-mgmt + client QoS |
ice_failure_ratio | client QoS + signaling |
mos_estimate distribution | Analytics (ClickHouse → Grafana) |
Control plane & infra
Standard RED per service (http_requests_total, error ratio, latency histograms), ws_connections_active per signaling pod, ws_resume_success_ratio, Kafka consumer lag, Redis/PG/CH saturation, node CPU/memory/network, pod restarts.
Dashboards (Grafana, provisioned from observability/dashboards/)
- Global overview — active users, rooms, join success/latency, per-region health.
- Media plane — SFU node heat (bandwidth/CPU/participants), TURN cluster usage, relay ratio, packet-loss/RTT distributions.
- Call quality — MOS distribution, loss/jitter/RTT percentiles by region/platform/tenant (ClickHouse-backed).
- Per-service RED — one per control-plane service.
- Data layer — PG replication lag, Redis cluster state, Kafka lag/ISR, ClickHouse ingest.
- On-call landing — the 10 signals that page, in one screen.
Alerting (Prometheus rules → Alertmanager)
Symptom-based pages, cause-based tickets:
| Page (P1/P2) | Condition (examples) |
|---|---|
| Join success ratio | < 99 % over 5 m (region) |
| Join latency | p95 > 5 s over 10 m |
| Media quality | packet loss p95 > 5 % or MOS p50 < 3.5 (region, 10 m) |
| WS resume failures | resume success < 90 % over 10 m |
| SFU capacity | fleet bandwidth or CPU > 80 % for 15 m (autoscaler headroom breach) |
| TURN health | healthy nodes < N-1 per cluster, or allocation error ratio > 1 % |
| Data layer | PG failover, replication lag > 30 s, Kafka lag growth sustained, Redis shard down |
| API errors | 5xx ratio > 1 % over 5 m per service |
Every alert links a runbook (docs/runbooks/, grown from game-days — 09 §4). SLO burn-rate alerting (multi-window) for the three SLOs defined in 12.
Synthetic Monitoring
Continuous black-box probes per region: scripted headless WebRTC client joins a canary room, publishes a tone, second probe asserts audio received + measures end-to-end latency; STUN/TURN allocate canaries against every coturn node. Synthetic failures page before customers notice — this is the primary detector for "everything is green but calls don't work."