Skip to main content

ADR 0040 — Scheduler Outbound Egress, SSRF Posture, and Secret Handling

Status: Accepted (security-engineer review gates the first shipped outbound call) Context date: 2026-08-06 Amended: 2026-08-17, after the Zero-Trust review of the scheduler. §1 previously described a Redis per-destination-host concurrency cap and a host-level circuit breaker that were never built; that text is replaced by what now exists (per-tenant and per-host rate caps in Redis, a per-schedule failure breaker in PostgreSQL) plus the destination-shape and inbound-limit rules the review required. §2 gains header-value sealing; §2a is new. Builds on: ADR 0032 (pkg/webhookdeliver), ADR 0025 §5 (redaction stance)

Context​

The scheduler makes outbound HTTPS calls to tenant-controlled URLs with tenant-controlled payloads on a tenant-controlled timetable — a strictly larger attack surface than the notification dispatcher, where the platform authors the payload. An attacker with a tenant account gets a free, periodic, internally-hosted HTTP client unless every request is constrained. Separately, each schedule carries an HMAC signing secret whose handling had to improve on the prior art (notification_db.webhook_subscriptions.secret is stored in plaintext).

Decision​

1. All guards live in the shared client, applied at two moments​

pkg/webhookdeliver (ADR 0032) is the single egress client for both notification and scheduler, and enforces:

  • Create/update time: public-host validation of the target URL (the existing dispatch.ValidateWebhookURL posture) — private, loopback, link-local, and metadata ranges rejected before a schedule can be saved.
  • Connect time: DNS re-resolution with IP pinning (the validated IP is the dialed IP — defeats DNS rebinding), no redirect following, HTTPS enforced in production, hard 10 s request timeout, bounded response read.
  • Destination shape: HTTPS on the scheme's default port only (the dev profile relaxes the port rule for local sinks, nothing else), and URLs embedding credentials (https://user:pass@host/) are refused. Without the port rule a schedule is an attributable port scanner, and the 443-only NetworkPolicy that would otherwise catch it does not exist in the single-box or compose profiles; without the credential rule, Go turns userinfo into an Authorization header, smuggling a credential past the reserved-header allow-list. Both are re-checked at delivery, so a target stored under older rules is not dialed.
  • Abuse bounds: payload ≤ 64 KiB; response body read ≤ 2 KiB then discarded (stored snippet control-char-stripped, per the ADR 0025 §5 redaction stance); no response-driven behaviour except the status code. Outbound rate caps per tenant and per destination host (Redis sliding window, services/scheduler/internal/ratelimit, counters only and reconstructible) bound how much traffic the platform will generate on a tenant's behalf and how much any single destination can be sent. A run held back by a cap is deferred, not failed: it costs no attempt and trips no breaker.
  • Inbound rate limits: POST /v1/schedules and POST /v1/schedules/{id}/trigger are limited per tenant in the scheduler itself, on the same Redis. The gateway's generic limiter keys on the token subject, and user_id is a free-text field on /v1/auth/token — a tenant rotating subjects has an unbounded budget there, so the per-tenant bound has to live in this service. Redis is a hard requirement for a meaningful limit: a per-process bucket enforces (replicas × limit).
  • Failure breaker: per schedule, in PostgreSQL — 100 consecutive failed runs, or 24 h with no success, transitions the schedule to disabled_failing. There is deliberately no host-level circuit breaker; the per-host rate cap covers the abuse case and a shared per-host breaker would let one tenant's misconfiguration mute another tenant's deliveries to the same provider.

One shared NetworkPolicy/egress rule covers both webhook-sending services — two call sites, one audited path.

2. Per-schedule secrets are envelope-encrypted at rest​

scheduler_db.schedule_secret stores the HMAC secret as AES-GCM ciphertext (nonce || ciphertext || tag) under a KEK delivered via External Secrets (ADR 0016), with the KEK version recorded in key_ref for rotation. The secret lives in its own table, not a schedule column, so no query on the aggregate can accidentally carry ciphertext into logs or API responses. The API never returns the secret after creation.

The same treatment applies to tenant-supplied header values (headers_ciphertext, migration 003). Because the allow-list bans Authorization, tenants carry their receiver credential in X-Api-Key and friends — those values are secrets in everything but name, and storing them in the schedule.target jsonb put them in plaintext beside non-secret configuration and echoed them from every GET and list. The target column now holds header names only (names decide which headers are sent and stay readable); values are sealed under the same KEK and resolved at delivery time. On update, a header submitted with an empty value keeps its stored value, so the read-modify-write round trip a names-only read implies cannot silently blank a credential.

2a. Failure detail is not a discovery oracle​

pkg/webhookdeliver distinguishes "the host does not resolve" from "the host resolves to an address we refuse to dial", and that distinction is a working enumeration oracle for internal service names (*.svc.cluster.local). Both the create/update rejection message and the run-history error class are therefore single, opaque, and identical across those causes — the run-history classes dns and blocked_ssrf are surfaced to tenants as one unreachable. The precise cause is kept where operators need it and tenants cannot read it: service logs and the delivery metrics.

3. Security review is a ship gate​

Because the surface is larger than notification's (URL and payload and timing are tenant-controlled), the security-engineer review of this posture is a gate before any outbound call ships — not a follow-up.

Alternatives Considered​

  • Relying on notification's existing guard by copy. Rejected: a copied SSRF guard diverges; the whole point of ADR 0032's extraction is one implementation, one audit target, one fix site.
  • Egress proxy as the only control. Rejected as sole control: a proxy centralises policy but does not see DNS-rebinding at the application dial, and it becomes a single choke point. The in-client guards are primary; a proxy remains compatible with them.
  • Plaintext secrets, matching webhook_subscriptions.secret. Rejected: matching prior art downward is not a justification. The scheduler sets the correct pattern; migrating notification's column to it is a recorded follow-up.
  • Platform-managed secrets only (no tenant-supplied). Rejected for v1 flexibility; tenants integrating existing receivers need to bring a secret. Storage handling is identical either way.
  • Validating the URL only at create time. Rejected: DNS answers change between create and fire — that gap is the rebinding attack. Connect-time re-validation with pinning is non-negotiable.

Consequences​

  • Positive: One egress client, one signature scheme, one SSRF review target across the platform; a fix lands once for both services.
  • Positive: Secret compromise via a database dump now also requires the KEK — a strict improvement, and the template for fixing notification's plaintext column.
  • Negative / accepted: IP pinning breaks exotic tenant setups (round-robin DNS with per-request rotation, redirects to CDNs). Documented limitation; correctness of the SSRF boundary wins.
  • Negative / accepted: Envelope encryption adds a KEK dependency to the delivery path; a KEK outage blocks signing new deliveries (fails closed). Accepted — the alternative is plaintext.
  • Negative / accepted: Header values are no longer readable after they are set; a tenant who loses track of a value must replace it. That is the same trade the HMAC secret already makes, and the reason an empty value on update means "keep".
  • Negative / accepted: Redis moves from "injected but unused" to a real dependency of this service. An outage fails open on the limits (scheduling and delivery continue unthrottled) rather than taking scheduling down; the limits are an abuse bound, not a correctness invariant.
  • Consequence: Mutating scheduler endpoints require the tenant_admin role (contract §Authorization), matching notification's webhook registration. Creating a schedule hands out a standing outbound-HTTP capability, which is not an end-user action.
  • Follow-up: Migrate notification_db.webhook_subscriptions.secret to the same envelope-encryption shape.
  • Follow-up (standing, raised stakes): The unauthenticated Kafka boundary (ADR 0025/0028/0030 follow-up) now also fronts schedule-events; SASL/mTLS with per-service produce ACLs remains the remediation track.