Skip to main content

ADR 0024 — Signup Notifications via Transactional Outbox and Email Channel

Status: Accepted Context date: 2026-07-26

Context​

ADR 0020 shipped self-serve tenant signup (POST /v1/signup) and left "event-driven provisioning — emit a tenant.provisioned event" as an explicit follow-up. A signup must now notify two parties: the platform owner (a new tenant exists) and the new tenant admin (welcome, here is your dashboard). Three gaps stood in the way:

  1. The platform had no email capability. No service could send mail at all.
  2. Auth had no Kafka producer. Auth is the only service that can observe a signup atomically, but it had never published an event.
  3. The notification service only did tenant webhooks. Its dispatcher consumes room-events and fans out to tenant-configured webhook endpoints — the wrong audience for platform-level lifecycle events.

Whatever ships must not make signup less reliable: a broker outage or SMTP failure must never fail or block POST /v1/signup.

Decision​

1. New platform-events Kafka topic for tenant lifecycle events​

Tenant lifecycle events get their own topic, platform-events, keyed by tenant_id so events for one tenant are ordered. The first event type is tenant.provisioned (source: auth) with data: tenant_id, tenant_name, tenant_slug, status, admin_user_id, admin_email, admin_display_name, signup_source.

Because the payload carries PII (the admin's email), the topic uses short retention (~7 days) rather than the long retention appropriate for operational event streams. Future lifecycle events (tenant.suspended, tenant.deleted) ride the same topic.

2. Auth publishes via a transactional outbox​

Auth adopts the same transactional-outbox pattern the room service already uses:

  • CreateTenantWithAdmin inserts the outbox row in the same transaction as the tenant and admin-user rows (services/auth/internal/store/store.go). Either the signup and its event both commit, or neither does.
  • A background relay (RelayOutbox) selects unpublished rows with FOR UPDATE SKIP LOCKED — safe across concurrent replicas — publishes them to Kafka, and marks them published_at. On broker trouble it records the attempt and retries next tick.
  • Signup therefore never fails or blocks on Kafka. KAFKA_BROKERS is optional for auth: when unset, the relay is disabled with a startup warning and signup works exactly as before.

3. Notification service owns email as a second delivery channel​

Email is a delivery channel, and the notification service is the delivery-channel service. It gains an email channel beside webhooks (services/notification/internal/email/):

  • A separate consumer group (notification-email) on platform-events, independent of the webhook dispatcher's room-events group.
  • SMTP via the Go stdlib net/smtp (STARTTLS, optional PLAIN auth) — no mail-provider SDK dependency.
  • Each tenant.provisioned event produces two independent sends: tenant_welcome to admin_email (login link built from PUBLIC_APP_URL) and owner_notify to PLATFORM_OWNER_EMAIL. Each has its own claim, retry budget, and DLQ parking, so one recipient failing never affects the other.
  • Per-recipient dedup uses email_deliveries claim rows with UNIQUE(event_id, kind) — the same idempotency pattern as webhook deliveries — so Kafka's at-least-once redelivery cannot double-send.
  • Bounded retry: 5 attempts with exponential backoff, then the raw event is parked on the email-dlq topic for inspection/redelivery. The consumer commits the offset only after both recipients are terminally resolved (sent, failed-to-DLQ, or deduplicated).
  • SMTP_HOST empty = email channel disabled; the consumer group simply does not start.

Security hardening applied at review (Zero-Trust posture):

  • Signup rejects control characters in org_name/display_name, and the emailer strips them again before mail bodies are built — free-text signup fields cannot inject mail-body lines or headers (the Subject was already Q-encoded).
  • The emailer re-validates every recipient (net/mail.ParseAddress, control-char scan) instead of trusting the Kafka payload; an invalid recipient is a terminal failure (recorded + DLQ'd), never an SMTP injection vector.
  • Plaintext SMTP is fail-closed: with STARTTLS off the service refuses to start unless SMTP_ALLOW_PLAINTEXT=true is set explicitly (dev/Mailpit only), and with STARTTLS on the send fails if the server does not advertise it (downgrade protection). SMTP_FROM is required whenever the channel is enabled; Helm/Kustomize defaults ship it empty so overlays must set a real, DMARC-aligned sender.
  • pkg/iplimit.ClientIP now trusts the right-most X-Forwarded-For hop (appended by our own ingress/LB) instead of the client-spoofable left-most one, so the per-IP signup limiter cannot be reset with a forged header.
  • Each consumed event is processed under a 60-second context timeout, so a slow or hostile SMTP endpoint cannot stall the consumer group indefinitely.

4. In-app notification feed in both dashboards​

The same platform-events stream drives a third, independent channel: a notification-inapp consumer group persists app_notifications rows (idempotent on UNIQUE(event_id, audience)), one per audience — platform (tenant_id NULL) and tenant (tenant_id set). The admin gateway exposes them at GET /admin/v1/notifications plus mark-read endpoints, deriving the audience predicate solely from the session (platform-operator session → platform; tenant JWT → tenant + own tenant_id; query parameters are never consulted), and both dashboards render a Notifications page with unread state. The in-app channel runs regardless of SMTP configuration — channels stay independent.

5. Dev and prod mail transport​

  • Dev: Mailpit in docker-compose (SMTP on 1025, web UI on 8025) — every developer can see the actual mails a signup produces, with zero external accounts.
  • Prod: any SMTP relay (SES, SendGrid, …) configured via Helm values, with SMTP credentials delivered through ExternalSecrets (ADR 0016).

Alternatives Considered​

  • Inline fire-and-forget publish from auth. Rejected: either events are silently lost during a broker outage, or signup fails on a Kafka error — both unacceptable. The outbox decouples signup durability from broker availability.
  • Reusing the room-events topic. Rejected: wrong ordering scope and wrong audience. The webhook dispatcher would fan tenant lifecycle events out to tenant-configured webhooks, leaking platform events to tenants.
  • IDs-only event with a callback to auth for the email address. Rejected: reintroduces synchronous coupling on the consume path — the notification service would depend on auth being up to process an event, defeating the point of the event.
  • A separate email microservice. Rejected: it would duplicate the consumer, dedup, and DLQ machinery the notification service already has, for no additional domain boundary. Email is a channel, not a domain.
  • Email-only notifications. Rejected after review: operators and tenant admins expect a visible notifications section in their dashboards; the in-app feed consumes the same topic, so it adds no coupling to auth.

Consequences​

  • Positive: Signup reliability is unchanged — the outbox commits with the signup transaction, and Kafka/SMTP failures degrade to delayed or parked notifications, never failed signups.
  • Positive: ADR 0020's event-driven-provisioning follow-up is now real infrastructure: billing and other future consumers subscribe to platform-events without touching auth.
  • Positive: Delivery semantics (idempotent claims, bounded retry, DLQ) are the proven webhook-dispatcher pattern applied to a second channel, not a new mechanism.
  • Negative / accepted: PII (the admin email) transits Kafka. Mitigated by the topic's short (~7-day) retention and its internal-only audience; no tenant-facing consumer reads it.
  • Negative / accepted: The per-replica outbox relay is at-least-once — SKIP LOCKED prevents concurrent double-relay, but a crash between publish and mark can replay an event. The consumer-side email_deliveries dedup absorbs this.
  • Follow-up: A NetworkPolicy for SMTP egress from notification pods is not yet in place; notification is currently the only pod that should open outbound SMTP connections.
  • Follow-up: app_notifications has no retention/pruning yet; a periodic purge of old read rows will be needed once volume grows.
  • Follow-up (security): Email-address verification (double opt-in) before the welcome mail — today the welcome is sent to whatever address the signup supplied, which combined with signup spam is an outbound-mail abuse surface. The in-process IP rate limiter should also move to the Redis-backed distributed limiter already planned for auth, plus a global outbound-send cap in the notification service.
  • Follow-up (security): Declare platform-events and email-dlq explicitly (short retention, produce ACL restricted to the auth relay) instead of relying on broker auto-creation; the DLQ carries the same PII as the source topic.

Addendum (2026-08-03): dashboard badge moved from polling to SSE push​

The dashboards' unread-badge polling (30s) was replaced by push: an AFTER INSERT trigger on app_notifications emits pg_notify with {audience, tenant_id}; the admin gateway holds one dedicated (non-pool) LISTEN connection per replica and fans out to per-client SSE streams at GET /admin/v1/notifications/stream (same session-derived audience scoping as the list endpoint). Transport notes: the SSE event is payload-free — clients refetch the scoped count endpoint, so the push channel adds no authorization surface; LISTEN has no replay, so every listener (re)connect broadcasts a synthetic refresh to all subscribers; streams send a comment ping every 25s and are capped at ~20 minutes so re-auth happens at reconnect through the normal middleware; the browser keeps the 30s poll as a live fallback whenever the stream is down. This is a transport change to the already-documented in-app feed — no new service, event type, or authz surface.