Skip to main content

12 — Production Best Practices

Operating standards the platform is built and run against. These are commitments, not aspirations — each is enforceable in CI, in review, or on-call.

SLOs (the contract everything serves)​

SLOTargetWindow
Join success ratio99.9 %30 d
Join latency p95< 5 s30 d
Established-call survival (no >3 s interruption from platform causes)99.9 %30 d
Control-plane API availability99.95 %30 d

Error budgets gate release pace: budget exhausted → feature freezes, reliability work only.

Release Engineering​

  • Trunk-based development; every merge produces an immutable, SHA-tagged image; deploys are Helm/kustomize changes in git (GitOps — cluster state is a repo, rollback is git revert).
  • Progressive delivery: staging soak with synthetic WebRTC probes → one prod region → all regions. Blue-green for gateway and SFU node pools; rolling with drain for everything else (08).
  • Feature flags for behavioral changes (per-tenant targeting); kill switches for risky subsystems (e.g., force-relay-off, recording-off).
  • Schema migrations: expand → migrate → contract, never breaking the running version; migrations run as pre-deploy Jobs, backward-compatible for one release either way.
  • Contract enforcement: OpenAPI/event/WS schemas in /contracts are CI-verified against implementations and against recorded fixtures; a breaking change fails the build, versioned types are the escape hatch (06).

Reliability Engineering​

  • Every synchronous call: timeout (default 2 s), bounded retries with jittered backoff only on idempotent operations, circuit breaker; every queue bounded; every cache with TTL. No unbounded anything.
  • Graceful degradation ladders documented per dependency (Redis down → joins still work, resume disabled; Kafka down → events buffer via outbox; PG down → established calls unaffected) — see 09 §3.
  • Load testing as CI cron: simulated-client harness (tools/) drives N-thousand concurrent joins against staging weekly; results tracked over time — regressions in join latency or SFU headroom are build failures, not surprises.
  • Chaos drills quarterly (node kills, region failout, PG failover) with measured RTO/RPO; runbooks updated from every drill and every incident.
  • Blameless postmortems within 5 working days of any SLO-impacting incident; action items tracked to closure.

Operational Hygiene​

  • On-call: symptom-based paging only (11), every page has a runbook, page rate is itself an SLO (< 2 pages/shift or the alerting is broken).
  • Capacity: media-plane utilization reviewed weekly; scale-up triggers at 60 % sustained (media capacity is bought in nodes and lead time exists); N+2 SFU headroom per region always.
  • Cost guardrails: per-tenant relay-bandwidth and recording metering from day one (it's the billing substrate later and the abuse detector now).
  • Data: retention policies enforced by TTLs, not cron scripts (ClickHouse TTL, Redis TTL, PG partitioning for sessions); PII inventory documented; deletion requests handled by tenant+user id across all stores.
  • Docs: architecture docs (this set) versioned with the code they describe; ADRs (docs/adr/) for every decision that reverses or extends one made here.

Engineering Standards​

  • One service = one owner team-of-record, one on-call rotation, one dashboard, one runbook index.
  • Code review required; internal/domain framework-free (hexagonal layering, 08); shared code only via pkg/ with the same review bar as services.
  • Tests: unit (domain logic), contract (against /contracts), integration (docker-compose per service), end-to-end (scripted WebRTC join in CI). Media-path E2E — two headless clients asserting actual audio through a real SFU + coturn — runs on every main merge.
  • Security: dependency + image scanning in CI, secrets scanning pre-commit, quarterly access review, annual external pentest of the media plane specifically (TURN abuse, SFU packet handling).

Phase 2 Preview (build order)​

With this architecture fixed, implementation proceeds in vertical slices, each independently shippable:

  1. Walking skeleton — Auth + Room + single-node LiveKit + single coturn + Web SDK: one room, real audio, deployed to staging via full CI/CD. (Proves the whole path end-to-end first.)
  2. Signaling service with resume + presence + events on Kafka.
  3. TURN management (multi-node, health, ephemeral creds) + media autoscaling.
  4. Moderation, roles, notification webhooks.
  5. Analytics ingest + ClickHouse + quality dashboards.
  6. Multi-region + DR drills + load-test hardening.
  7. Recording, then billing substrate.