09 — Scalability, High Availability & Disaster Recovery
1. Scalability Model
Stateless services
Every control-plane service is stateless: no session affinity required for REST; any replica serves any request. Session-shaped state lives in Redis (presence, signaling sessions) or PostgreSQL (durable). Consequence: scaling is purely replicas: N, and instance death loses nothing but in-flight requests.
Sticky sessions exist in exactly one place, by nature not by design debt: a WebSocket lives on one Signaling instance and a room's media lives on one SFU node. Both are engineered to make that stickiness cheap to break — WS resume via Redis-backed session state (05), room migration via ICE restart. We do not use LB-level sticky sessions anywhere.
Fan-out fabric
- Redis Pub/Sub — intra-region realtime fan-out: signaling events cross instances in
<1 ms. Per-room channels bound fan-out cost to room size. - Kafka — inter-service and cross-region event streaming: durable, replayable, consumer-lag-observable. Realtime path never blocks on Kafka.
Horizontal scaling — what triggers what
| Tier | Autoscaling signal (HPA/custom) | Notes |
|---|---|---|
| Gateway | RPS per pod, p99 latency | 3 → 30 pods |
| Auth / Room / TURN-mgmt / Presence | CPU + RPS | cheap, over-provision floor of 3 |
| Signaling | concurrent WS connections per pod (custom metric, target ~10 k/pod) | scale-out early; scale-in only via drain |
| Notification / Analytics ETL | Kafka consumer lag (KEDA) | lag-driven, scales to floor when idle |
| SFU | bandwidth + CPU + participant count per node (custom autoscaler → cluster-autoscaler node pool) | new node joins LiveKit mesh automatically; scale-in = drain node |
| Coturn | allocation count + egress bandwidth per node (fleet ASG) | credentials are stateless → adding/removing nodes is just issuance-list changes |
| PostgreSQL | read replicas for read-heavy endpoints; primary scales vertically | control-plane writes are small; partition by tenant later if ever needed |
| Redis | Redis Cluster reshard | keyspace designed with room/user hash tags |
| Kafka | partition count headroom (start 12/topic) | consumers scale within partition count |
| ClickHouse | add shards; ingestion is batched |
Capacity anchors (voice-only, Opus ~48 kbps + overhead)
- One SFU node (16 vCPU, 10 Gbps): ~5–8 k concurrent audio subscriptions comfortably.
- 1 M concurrent users ≈ ~150–200 SFU nodes globally + TURN for ~20 % of egress — a few dozen coturn VMs per major region. Control plane at that scale is small: ~100 signaling pods dominate; Redis/Kafka sized accordingly. Nothing in the architecture changes between 1 k and 1 M users — only counts.
Regional & global
- Regional TURN and SFU clusters in every serving region — media latency is physics; the media plane must be near users.
- A room is pinned to one SFU region; cross-region participants reach it over the backbone (LiveKit multi-node room spanning is a later optimization).
- Global load balancing: GeoDNS/anycast routes REST+WS to nearest healthy region; TURN Management returns region-local ICE servers; health-based failout at DNS + LB layers.
- Control-plane truth: PostgreSQL primary per home region with async cross-region replica; Kafka per-region clusters with MirrorMaker for the few globally-consumed topics; Redis strictly region-local.
2. High Availability
| Layer | Mechanism | Target |
|---|---|---|
| Services | ≥3 replicas across 3 AZs, PDBs, anti-affinity | zero-impact single-pod/AZ loss |
| PostgreSQL | streaming replication, automated failover (operator/managed), synchronous standby in-region | RPO 0 in-region, failover < 30 s |
| Redis | Redis Cluster, replicas per shard, AZ-spread | failover < 10 s; data is reconstructible anyway |
| Kafka | RF=3, min.insync.replicas=2, rack-aware | broker loss transparent |
| ClickHouse | 2 replicas per shard | analytics is not on the call path |
| SFU | N+2 node headroom per region | room re-host < 3 s |
| Coturn | ≥3 nodes/region behind issuance-side selection | failed node dropped from issuance in seconds |
| Regions | full media plane per region; control plane active-active for reads/joins, single-home writes | region loss → degraded-but-serving |
Availability targets: control-plane API 99.95 %; media plane (established calls surviving) 99.9 %; join success 99.9 %.
3. Failure Recovery Playbook
| Failure | Detection | Recovery | User impact |
|---|---|---|---|
| TURN node dies | TURN-mgmt active probes (STUN + allocate canary, 5 s interval) | node removed from issuance; existing allocations on it break → clients ICE-restart onto surviving nodes with fresh creds | relay users: 1–3 s audio gap; direct-UDP users: none |
| SFU node dies | LiveKit node liveness (Redis registry) + Prometheus | rooms re-assigned to healthy node; Signaling pushes ice_restart; clients renegotiate to new node | all participants of hosted rooms: ~2–3 s gap; roster/state intact (control plane owns it) |
| Signaling instance dies | LB health check + heartbeat gap | clients reconnect (backoff), resume on any instance via Redis session state; missed frames replayed | sub-second blip; media unaffected (media ≠ signaling path) |
| Redis failure (shard) | cluster gossip + sentinel metrics | replica promotion; presence/session data is TTL-ephemeral — worst case clients fall back to full rejoin instead of resume | brief presence staleness; possible forced rejoins |
| Redis failure (total, region) | — | services degrade deliberately: joins still work (PG authoritative), resume disabled, rate limits fail-open with conservative static limits, roster rebuilt from SFU state | degraded UX, calls continue |
| PostgreSQL primary loss | operator/managed failover | standby promotion < 30 s; services retry with backoff; reads unaffected (replicas) | joins/mutations stall ≤ 30 s; established calls unaffected (media plane has no PG dependency) |
| Kafka outage | broker health, producer errors | producers buffer (bounded) + outbox pattern for critical events (see 06); consumers catch up on recovery — everything async by design | webhooks/analytics delayed; calls unaffected |
| K8s node failure | node-controller | pods rescheduled; PDB+anti-affinity guarantee remaining capacity; SFU node = SFU-failure path above | per-tier impact as above |
| Full region failure | global health checks | GeoDNS fails region out; clients rejoin → rooms re-pin to nearest region; PG replica promoted if home region lost (RPO ≈ seconds, async) | in-region calls drop and reconnect cross-region; global joins continue |
Design invariant behind every row: an established call must survive any single control-plane failure, because the media plane's only hard runtime dependencies are itself and (for relay users) Coturn.
4. Disaster Recovery
- Backups: PostgreSQL — continuous WAL archiving + nightly base backups, 30-day PITR, cross-region bucket replication. ClickHouse — daily snapshots to object storage. Kafka — critical topics mirrored cross-region; Redis — none (ephemeral by contract). Recordings — object storage with cross-region replication + lifecycle policies.
- Objectives: RPO — 0 in-region (sync standby), ≤ 30 s cross-region (async). RTO — single-component: seconds–minutes (automated, above); full region: ≤ 15 min to serve all traffic from surviving regions (automated DNS failout + promotion runbook); catastrophic (restore from backup): ≤ 4 h.
- Infra as code: entire stack reproducible from
deploy/terraform+ Helm — a region can be rebuilt from scratch + backup restore without tribal knowledge. - DR drills: quarterly game-days — kill an SFU node, a PG primary, and a full staging region; measure real RTO/RPO against targets; runbooks live in the repo and are updated from every drill.