Skip to main content

ADR 0001 — Multi-Region Deployment Strategy

Status: Accepted (Phase 2, slice 6) Context date: 2026-07

Context​

The platform must serve users worldwide with sub-150ms media latency and survive the loss of an entire region (docs/09). Media latency is physics — the SFU and TURN relay must be near the user — while the control plane's source-of-truth data (identity, rooms) benefits from consistency. These two pull in opposite directions, so we split the strategy by plane.

Decision​

1. Media plane is region-local and room-pinned​

Every serving region runs a full media plane: an SFU cluster (LiveKit) and a TURN/STUN cluster (Coturn). A room is pinned to exactly one SFU region at creation (rooms.media_region), chosen by creator locality or tenant policy. All of a room's participants converge on that region's media cluster; cross-region participants reach it over the backbone. This keeps the common case (participants near each other) optimal and makes the media failure domain a single region.

Rationale for room-pinning over room-spanning: a room spanning SFU nodes across regions (cascading SFUs) is a real LiveKit capability but adds inter-node forwarding latency and operational complexity we defer until multi-continent single rooms are a proven need. The media_region column and region-aware token/credential issuance already in the code are the forward-compatible seam.

2. TURN issuance is region-aware, credentials are stateless​

The TURN management service returns an iceServers list ordered nearest-region-first with a fallback cluster (REGION config + client geo). Because Coturn credentials are stateless HMAC (validated offline), adding/removing regional TURN capacity or failing a region out requires only changing what we issue — no shared credential state to replicate.

3. Control-plane data: single-writer home region, read replicas elsewhere​

  • PostgreSQL — one primary in a home region per tenant with asynchronous cross-region read replicas. Writes (create room, moderation) are single-homed; reads (room lookup on join) can be served regionally. RPO on home-region loss is seconds (async replication); the runbook promotes a replica.
  • Redis — strictly region-local. Presence, signaling sessions, and rate-limit counters are ephemeral and reconstructible; we never replicate Redis cross-region. Global presence queries aggregate across regions via the Presence API, not via cross-region Redis.
  • Kafka — per-region clusters. The few globally-consumed topics are mirrored (MirrorMaker) so notification/analytics can run per-region or centrally.
  • ClickHouse — regional ingest, with a central rollup for global analytics (analytics is never on the call path, so its consistency needs are loose).

4. Routing​

GeoDNS / anycast routes REST and WebSocket traffic to the nearest healthy region. Health-based failout operates at both the DNS and load-balancer layers. The signaling signal_url returned at join already carries a regional host, so a client is steered to the region hosting its room.

5. Failure domains and the core invariant​

The invariant from docs/09 holds per region: an established call survives any single control-plane failure, because the media plane's only hard runtime dependencies are itself and (for relay users) Coturn. A full region loss drops that region's in-progress calls; clients reconnect and their rooms re-pin to the nearest surviving region. Global joins continue throughout because the control plane is active-active for reads and single-home only for writes.

What is implemented now vs. deferred​

Implemented (code-level, single-region-runnable):

  • REGION config across turn-mgmt and analytics; region carried into TURN issuance and QoS rows.
  • rooms.media_region pinning column and region field in TURN responses.
  • Per-service /readyz readiness so a region's load balancer can fail a degraded pod (or region) out of rotation without restarting it (this slice).
  • Graceful degradation ladders (this slice): join survives Redis loss in media-only mode; TURN loss degrades to host/srflx candidates.

Deferred (infra, needs a real multi-region cluster):

  • Actual second region, GeoDNS, cross-region PostgreSQL replication, Kafka MirrorMaker. These are Terraform/Helm concerns validated in a cloud environment, not on a single Docker host.
  • Cross-region room spanning (cascading SFUs).

Consequences​

  • Simple, well-understood failure domains: one region = one media failure domain; one home region = one write failure domain per tenant.
  • Cross-region participants in a pinned room pay a backbone hop — acceptable for the common near-each-other case, revisited if global single-rooms become common.
  • Operational cost: N full media planes. Mitigated because media capacity is bought in nodes and scales independently (docs/09).

Verification​

Region-awareness and degradation are exercised by tools/chaos.mjs (DR drills: TURN/Redis/Postgres/signaling failure → documented recovery) and tools/loadtest.mjs (join-path SLO under concurrency). True multi-region failover is a cloud game-day, scripted from this ADR's runbook when a second region exists.