02 — Services & Interactions

Service Catalog

Service	State	Store	Scaling signal	Sync dependencies
API Gateway	Stateless	Redis (rate limits)	RPS, p99 latency	Auth (JWKS cache)
Authentication	Stateless	PostgreSQL	RPS	—
Room	Stateless	PostgreSQL, Redis	RPS	Auth (token introspection via JWT), SFU coordinator API
Signaling	Stateless*	Redis pub/sub	Concurrent WS connections	Room, Presence
TURN Management	Stateless	Redis	RPS	— (Coturn validates offline via shared secret)
Presence	Stateless	Redis	Ops/sec	—
Notification	Stateless	Kafka offsets	Consumer lag	Tenant webhooks (outbound)
Analytics	Stateless	ClickHouse, Kafka	Ingest rate, consumer lag	—
Scheduler	Stateless	PostgreSQL, Redis	Runs-due backlog (scheduler_runs_due_backlog)	Tenant webhooks (outbound)
SFU (LiveKit)	Stateful per session	In-memory + Redis (node registry)	Bandwidth, CPU, tracks	Redis
Coturn	Stateful per allocation	—	Bandwidth, allocations	—
Recording	Stateful per job	Object storage	Concurrent jobs	SFU egress API

* Signaling holds live WebSocket connections (connection-stateful) but all session state lives in Redis, so any instance can serve any reconnecting client.

---

1. Authentication Service

Responsibilities: user authentication, JWT issuance (access + refresh), role management, room authorization grants, API-key auth for BaaS tenant servers.

Design points:

• Two token classes. (a) User tokens — RS256/EdDSA-signed JWTs, 15-min access + 30-day rotating refresh (refresh tokens are opaque, hashed at rest, single-use with reuse detection). (b) Tenant API keys — long-lived key/secret pairs for server-to-server calls (e.g., a customer's backend minting room tokens for its own users).
• Asymmetric signing so every service (gateway, room, signaling, SFU token validation) verifies locally against a published JWKS — no network hop per request. Key rotation via kid with a two-key overlap window.
• Permission model (see 03 — schema): platform roles (tenant_admin, user) plus per-room roles (moderator, speaker, listener) carried as claims in the room grant, not the identity token — room permissions change without re-authenticating.

Deliverables satisfied: REST APIs (04), JWT strategy (10), permission model (03).

2. Room Service

Responsibilities: room CRUD and lifecycle, join/leave, metadata, persistence, participant roles (moderator/speaker/listener), mute state, moderation actions.

Design points:

• Room lifecycle: created → active (first participant) → idle (last participant left) → closed (explicit, tenant suspension, or inactivity). Closed rooms are retained as historical records; "delete" is soft-delete for audit, hard-purge by retention policy.
  • The idle transition happens in the same transaction that ends the last live session (/v1/rooms/{id}/leave and signaling's /internal/sessions/end share it), stamping rooms.idle_since. Rejoining clears idle_since and returns the room to active. Both sides take the room's row lock, so a join can never interleave with the emptiness check.
  • The inactivity reaper is a ticker loop (ROOM_REAPER_INTERVAL, default 5m) running on every room replica. Each tick it claims a bounded batch (200) of expired rooms with FOR UPDATE SKIP LOCKED — the same claim pattern the Scheduler's planner uses — and closes them: idle rooms empty for longer than ROOM_IDLE_TTL (default 24h) with reason idle_timeout, and never-joined created rooms older than the same window with reason abandoned. The close re-asserts every predicate plus "no live sessions" at commit time, so a room that gained a participant mid-tick is skipped. room.closed goes out through the existing outbox.
  • Why reap at all: an empty room costs nothing at runtime (no signaling goroutine, no Redis subscription, no SFU allocation). The reason is slug reuse — the unique index on (tenant_id, slug) WHERE deleted_at IS NULL means an abandoned room squats its slug against the tenant forever.
• Closing tears down the media room: every close path (explicit delete, tenant-suspension sweep, reaper) calls DeleteRoom at the SFU after the database close commits, so a closed room cannot leave participants connected and still streaming. The SFU call is best-effort: PostgreSQL is authoritative, and LiveKit reaps empty rooms on its own timer.
• Media coordinator lives here: on first join, the Room Service asks the SFU cluster (LiveKit does node assignment natively via its Redis-backed routing) for the node/region hosting the room, and mints the media access token embedding room grants (canPublish for speakers, canSubscribe for all, moderator flags as metadata).
• Moderation is control-plane-authoritative: mute/kick/promote are Room Service API calls that (1) update PostgreSQL/Redis state, (2) instruct the SFU (mute track / remove participant via server API), (3) emit a Kafka event that fans out to clients via Signaling. A "muted by moderator" client cannot unmute itself because its publish grant is revoked at the SFU, not merely hidden in UI.

3. Signaling Service

Responsibilities: WebSocket lifecycle, SDP offer/answer relay, ICE candidate exchange, connection state, heartbeats, reconnection, participant synchronization.

Design points:

• Stateless-with-registry: each instance holds live sockets; a Redis registry maps session_id → instance and per-room pub/sub channels fan messages out across instances, so two participants of one room can be connected to different signaling pods.
• Protocol & state machine: full message catalog and client/server state machine in 05.
• Division of labor with the SFU: clients negotiate SDP with the SFU (it is the peer); Signaling is the authenticated, ordered transport for that negotiation plus room-level events (participant list sync, active-speaker updates, moderation notices). With LiveKit, the SFU's own signaling handles SDP; our Signaling Service wraps/proxies it so clients see one protocol and one auth model, and so we can swap SFU engines later without breaking clients.

4. TURN/STUN Management Service

Responsibilities: temporary TURN credentials, ICE server configuration issuance, credential expiry, region/cluster selection, Coturn health monitoring, usage metrics. TURN itself is Coturn — never reimplemented.

Design points:

• Credential scheme: Coturn's TURN REST API long-term-credential mechanism — username = <unix_expiry>:<user_id>, password = base64(HMAC-SHA1(shared_secret, username)). Credentials are minted by this service and validated by Coturn offline against the same secret: zero runtime coupling, TTL 10 minutes (covers ICE + restarts; re-fetched on renegotiation).
• Region/cluster selection: the service returns an ordered iceServers list — nearest healthy TURN cluster first (client-geo via edge headers, room's pinned region as tiebreak), one fallback cluster second. STUN servers (same Coturn fleet, no auth needed) listed first so relay is only used when required.
• Health & failover: active probing (STUN binding + TURN allocate canary per node), nodes failing probes are removed from issuance within seconds. Because credentials are stateless, failover requires nothing but changing what we hand out.
• Metrics: allocation counts, relayed bytes per cluster/tenant (from Coturn's Prometheus exporter + its Redis status DB) feed capacity planning and (future) billing.

5. SFU Service

SFU Selection

Recommendation: LiveKit (self-hosted, Apache-2.0).

Criterion	LiveKit	mediasoup	Janus	Pion (raw)
Level of abstraction	Complete SFU server with clustering	C++/Node library — you build the server	Server + plugins (C)	Go toolkit — you build everything
Horizontal scaling / multi-node	Built-in (Redis-based node routing, room pinning)	DIY	DIY (complex)	DIY
Client SDKs (Web/Android/iOS/Desktop)	First-party, mature, all four	Web official; mobile community	Community, uneven	None
Simulcast / SVC, adaptive subscription	Built-in	Supported (you orchestrate)	Partial	You implement
Active speaker detection	Built-in	Audio levels exposed; you aggregate	Plugin-dependent	You implement
Recording hooks	Egress service included	DIY	Plugin	DIY
Server API + webhooks	Yes (room/participant control)	You design it	Admin API (limited)	You design it
Kubernetes story	Helm charts, production-documented	DIY	DIY	DIY
Language / ops profile	Go (single binary, easy ops)	C++ addon inside Node	C	Go

Reasoning: mediasoup and Pion are excellent building blocks, but choosing them means building and hardening our own clustering, node assignment, reconnection semantics, SDKs for four platforms, and recording — precisely the undifferentiated heavy lifting this platform should not spend its first year on. Janus's plugin model and C codebase carry higher operational risk with weaker official mobile SDKs. LiveKit (itself built on Pion) gives us a production-proven distributed SFU with the exact feature list this project requires (simulcast, active-speaker, bandwidth estimation, reconnection, egress/recording, Prometheus metrics) while remaining self-hosted and replaceable — our Signaling wrapper and token-minting seam isolate clients from the engine choice. If we later need packet-level custom behavior, Pion (LiveKit's own foundation) is the escape hatch.

Responsibilities (as deployed): audio forwarding, producer/consumer management, simulcast/adaptive subscription, active speaker detection (fanned out through Signaling), congestion control (REMB/transport-cc), ICE restarts and reconnection, egress hooks for recording, Prometheus metrics.

6. Presence Service

Tracks online users, room occupancy, speaking/muted/idle state — in Redis with TTL-based liveness (see 03 for keyspace). Sources of truth: Signaling heartbeats (online/idle), SFU webhooks (speaking), Room Service (muted). Serves read APIs (who's in room X, is user Y online) and emits occupancy-change events to Kafka. Everything is TTL-expiring: a crashed client disappears from presence within one missed heartbeat window, no cleanup jobs needed.

7. Notification Service

Pure Kafka consumer → dispatcher. Consumes domain events (user.joined, user.left, room.closed, moderation.muted, …see 06) and dispatches to: (a) tenant webhooks (HMAC-signed, retried with exponential backoff + DLQ), (b) mobile push (APNs/FCM) for invites/room-start, (c) internal listeners. In-room real-time delivery is not its job — that's Signaling via Redis pub/sub; Notification handles everything that leaves the platform or targets users not currently connected.

8. Analytics Service

Two ingest paths: (a) client SDKs POST batched WebRTC stats (getStats() snapshots: packet loss, RTT, jitter, bitrate) every 10 s; (b) Kafka ETL of domain events and SFU/Coturn metrics. Writes land in ClickHouse in batches (native async insert). Computes per-session MOS (E-model from loss/jitter/RTT) at session close. Serves tenant-facing quality dashboards and internal capacity analytics. Schema in 03.

9. API Gateway

Terminates TLS, verifies JWTs (local JWKS), enforces per-user/per-tenant/per-IP rate limits (Redis sliding window), validates request shapes (OpenAPI), routes to services, injects OpenTelemetry trace context, emits structured access logs. WebSocket upgrade for /signal is authenticated at the gateway then proxied to Signaling. Implementation choice: an off-the-shelf cloud-native gateway (Envoy-based, e.g. Envoy Gateway or Kong) — like TURN, not something to build from scratch.

10. Scheduler Service

Responsibilities: tenant-facing cron-as-a-service — the Schedule aggregate (cron/interval/one-shot triggers with IANA timezones), occurrence computation, the JobRun lifecycle, webhook/event delivery with retry + DLQ, per-schedule failure breaker with auto-disable, and the tenant-facing run-history/delivery-log API. Port 8089, database scheduler_db, gateway route prefix /v1/schedules/ (OpenAPI: contracts/openapi/scheduler.yaml).

Design points:

• One binary, three loops (api, planner, executor), each toggleable by env flag — one Deployment initially, executors split out by values change when webhook fan-out dominates. All replicas run the same loops: leaderless, coordinated only by FOR UPDATE SKIP LOCKED claims and a UNIQUE (schedule_id, scheduled_for) occurrence index (ADR 0033). Semantics: exactly-once run materialisation, at-least-once delivery with a stable X-Idempotency-Key (ADR 0034).
• Two actions per schedule: webhook — the scheduler delivers directly to the schedule's own URL through the shared pkg/webhookdeliver client (same SSRF guard, HMAC signature scheme, and backoff as the notification dispatcher — ADR 0032/0040); event — it emits schedule.triggered, which notification fans out to the tenant's subscription endpoints.
• Events produced: schedule-events (key tenant_id, 14 d) and schedule-dlq — lifecycle + schedule.triggered + run.failed/disabled_auto via a transactional outbox in scheduler_db, high-volume run.started/succeeded/skipped best-effort (06, ADR 0038).
• Events consumed: platform-events twice — group scheduler-tenant-status (suspension projection, ADR 0025 pattern; suspension windows are never backfilled, ADR 0036) and group scheduler-entitlements (ADR 0030 entitlement snapshot + quota verdicts; over-quota occurrences are skipped(reason=quota_exhausted), ADR 0039).
• Storage: PostgreSQL is the sole source of truth (schema in 03); Redis holds only reconstructible concurrency buckets and per-host breakers. Autoscaling keys on the runs-due backlog, not CPU.

---

Service Interaction Matrix

From ↓ To →	Auth	Room	Signaling	TURN Mgmt	Presence	SFU	Kafka
Gateway	REST (login, refresh)	REST (rooms)	WSS proxy	REST (credentials)	REST (queries)	—	—
Room	JWT verify (local)	—	publish room events via Redis	—	occupancy read	server API: token mint, mute, kick	produce
Signaling	JWT verify (local)	join validation (REST)	Redis pub/sub (cross-instance)	—	heartbeat writes	proxy SDP/ICE session	produce
Presence	—	—	—	—	—	—	produce (occupancy)
SFU	—	webhooks (participant/track events)	—	—	—	—	— (webhooks → Room → Kafka)
Notification	—	—	—	—	—	—	consume
Analytics	—	—	—	—	—	scrape metrics	consume
Scheduler	JWT verify (local)	—	—	—	—	—	produce (schedule-events), consume (platform-events)

Rule of thumb enforced by this matrix: synchronous calls only on user-blocking paths; everything else is an event.