Skip to main content

ADR 0025 — Tenant Suspension: Event-Driven Enforcement and Notification

Status: Accepted Context date: 2026-07-27

Context​

The ops console has had a suspend/activate action for tenants since the admin gateway shipped, but it was a paper tiger: Suspend only flipped tenants.status in auth_db. Nothing read that flag —

  1. Login kept working. Users of a suspended tenant could still sign in and mint fresh tokens.
  2. Existing tokens kept working. Access tokens live ~1 hour; nothing between token issue and expiry consulted tenant status.
  3. Rooms kept running. The room service never sees auth_db (every service owns its own database), so suspended tenants could create rooms, join calls, and stream media indefinitely.
  4. The tenant was never told. A suspended admin just saw requests start failing (or not), with no reason and no channel — they are locked out of the dashboard the moment enforcement exists.

ADR 0024 built the pieces this needs: the platform-events topic (which explicitly reserved tenant.suspended as a future event type), auth's transactional outbox and relay, and the notification service's email and in-app channels.

Decision​

1. The gateway emits lifecycle events through auth's outbox — a second writer, on purpose​

setTenantStatus in the admin gateway (admin/gateway/internal/tenants/tenants.go) now runs one auth_db transaction: the status UPDATE, a lookup of the tenant's active tenant_admin emails, and an INSERT of a tenant.suspended / tenant.activated envelope (source admin-gateway, key tenant_id) into the same outbox table that auth's RelayOutbox drains onto platform-events. The status change and its event are atomic — if any step fails, both roll back and the request 500s.

This makes the gateway and the auth service two writers of one outbox contract. That coupling is deliberate and documented rather than accidental: the gateway already reads and writes auth_db tables directly (tenants, users) as its established pattern, and the outbox row format (id, topic, key, payload with an ADR 0024 envelope) is the narrow, stable surface being shared. Admin emails are captured in the event payload (admin_emails) so consumers never have to read auth_db themselves — the same IDs-plus-data stance ADR 0024 took for tenant.provisioned. The suspend endpoint requires an operator-authored reason, which rides the payload into every tenant-facing message.

2. Auth enforces synchronously at every credential-issuing path​

Auth reads tenants.status in its own database, so it enforces with zero staleness at the three places credentials are born:

  • Login (POST /v1/auth/login) — a user of a suspended tenant gets 403 code tenant_suspended ("your organization is suspended; contact support").
  • Refresh (POST /v1/auth/refresh) — same rejection; a live session dies at its next refresh.
  • API-key token mint (services/auth/internal/api/token_mint.go) — server SDKs of a suspended tenant can no longer mint access tokens.

After suspension, no new credential of any kind is issued. What remains is the window of already-issued access tokens (~1 hour TTL) — that is the room-side gate's job.

3. Room service: an event-driven tenant_status projection gates every API call​

The room service must not call auth per request (sync coupling on the hot path) and must not read auth_db (database-per-service). Instead it keeps a local projection:

  • A new consumer group room-tenant-status on platform-events (services/room/internal/tenantstatus/) upserts room_db.tenant_status (migration 006_tenant_status.sql). The upsert is guarded by event_occurred_at (WHERE excluded.event_occurred_at >= tenant_status.event_occurred_at), so a redelivered or reordered older event can never un-suspend a tenant.
  • withAuth in the room API runs a blanket check: every tenant-scoped call from a suspended tenant is rejected with 403 code tenant_suspended. This catches unexpired access tokens within seconds of the operator clicking Suspend.
  • The gate is deliberately fail-open, documented in code: a missing row (no lifecycle event seen yet for that tenant) means active, and a DB error during the check logs a warning and allows the request. Availability wins over enforcement for transient blips — auth already refuses new tokens, so this gate only shortens the window for already-issued ones.

4. Suspension sweeps live rooms closed and tears down media​

Blocking new calls is not enough — a suspended tenant's in-progress calls would otherwise run until the participants hang up. On tenant.suspended, the same consumer:

  • Closes every active room of the tenant through the existing internal close path (store.CloseRoom, reason tenant_suspended) — the same path DELETE /v1/rooms/{id} uses, so room.closed events are emitted and tenant webhooks fire as usual. ErrNotFound (already closed, concurrently or on redelivery) is skipped, keeping the sweep idempotent.
  • Best-effort deletes each LiveKit room via Twirp DeleteRoom, hard-disconnecting live participants immediately. A 404 is the expected idempotent no-op; any other SFU error is logged and swallowed — the DB close plus the withAuth gate are the source of truth, and a participant of an undeleted room is cut off at their next signaling round-trip anyway.

Signaling tickets and TURN credentials are only ever minted through room APIs, which the withAuth gate now blocks — so no separate revocation mechanism is needed for them.

5. Notification: email to the tenant's admins, plus an in-app record​

A suspended admin cannot log in, so email is the only channel that reaches them. The notification service's emailer (services/notification/internal/email/emailer.go) handles both event types with kinds tenant_suspended / tenant_activated:

  • One multi-recipient SMTP send to all admin_emails from the payload, with one claim per (event, kind) in email_deliveries — the ADR 0024 dedup, retry (×5 with backoff), and email-dlq machinery, unchanged. Each recipient is re-validated before send; invalid addresses are dropped individually rather than failing the delivery.
  • The suspension email includes the operator's reason; the reactivation email announces the tenant is active again.
  • The in-app feed writes a tenant-audience app_notifications row (title, body with reason, control-characters stripped and length-capped) — visible in the tenant dashboard after reactivation, so the record of what happened and why survives the lockout. No platform-audience row is written: the ops audit log already records suspend_tenant / activate_tenant with the acting operator, and a duplicate feed entry would add noise, not information.

Alternatives Considered​

  • Gateway calls an auth API to emit the event. Rejected: an extra synchronous hop, and — worse — non-atomic: the status flip and the event could diverge if the call fails after the update (or vice versa). Sharing the outbox table inside one transaction is less coupling than it looks, because the gateway already writes these auth_db tables directly.
  • Room service checks auth per request. Rejected: synchronous coupling on the hottest path in the system, and an auth outage would take the room API down with it. The projection costs one indexed local read.
  • Tenant status as a JWT claim. Rejected: stale for the full token TTL (~1 hour) — exactly the window that needed closing. It also bakes a mutable fact into an immutable credential.
  • Block new joins but let live rooms run out. Rejected: suspension is an operator action against a misbehaving tenant; live sessions outliving it (potentially for hours) defeats the purpose. The sweep reuses the existing close path, so it cost little.
  • Revoke signaling tickets and TURN credentials separately. Not needed: both are only obtainable through room APIs the blanket gate now rejects, and both are short-lived by design.

Consequences​

  • Positive: Suspension is now real. New credentials stop instantly (auth, zero staleness); existing tokens stop working against rooms within seconds; live calls are torn down; the tenant is told why.
  • Positive: platform-events fulfils its ADR 0024 promise — tenant.suspended/tenant.activated ride the existing topic, outbox, email, and in-app machinery with no new infrastructure.
  • Positive: The occurred_at guard plus idempotent close-sweep make the projection safe under Kafka's at-least-once, out-of-order reality.
  • Negative / accepted: Enforcement propagation to the room service is seconds, not instant (relay tick + Kafka + consumer lag). Acceptable for an operator-initiated action; the synchronous auth gate bounds the blast radius to already-issued short-TTL tokens.
  • Negative / accepted: Admin emails transit the internal platform-events topic — the PII stance already accepted in ADR 0024 (short ~7-day retention, internal-only audience).
  • Negative / accepted: The gateway is a second writer of auth's outbox — a documented contract, but schema changes to that table now have two clients.
  • Negative / accepted: The fail-open projection means a tenant suspended before the room consumer ever ran (or during a projection outage) slips through until the event arrives — bounded by at-least-once redelivery, which keeps retrying until the upsert commits.
  • Negative / accepted (security review): A participant holding an unexpired media token (10-minute TTL) can reconnect to LiveKit directly after the sweep, because LiveKit auto-creates rooms for tokens with a join grant — a media-only bypass of the withAuth gate, bounded to ≤10 minutes and to users already in the tenant's rooms. Accepted for this slice.
  • Follow-up (security): Close the media-token window: create LiveKit rooms explicitly at join and disable server-side auto-create, and feed tenant.suspended to the signaling service so open WebSockets are closed rather than left to drain.
  • Follow-up (security): The Kafka boundary is unauthenticated (plain TCP, auto-topic-creation) — an in-network producer could forge lifecycle events (fake tenant.activated, mass-close via fake tenant.suspended, spam via admin_emails). Enforce SASL/mTLS with per-service produce ACLs on platform-events (MSK supports both), and NetworkPolicy-restrict broker reachability until then. Pre-existing boundary; this slice raises its blast radius.
  • Follow-up: The projection has no reconciliation/backfill job against auth_db; a periodic sweep would close the fail-open gap for events that predate the consumer and would catch any drift.