Skip to main content

ADR 0037 — Deterministic Jitter and the scheduled_for / fire_after Split

Status: Accepted Context date: 2026-08-06 Builds on: ADR 0033 (claim-queue scheduling), ADR 0034 (idempotency key)

Context​

Cron traffic is pathologically synchronized: a large fraction of all schedules land on :00. Without spreading, every top of the hour is a thundering herd against the planner, the executor pool, and the network egress path. But naive random jitter breaks the tenant's exactness model — the idempotency key, the signed payload, and the run history must all agree on when the occurrence was, and replays/debugging must be reproducible.

Decision​

1. Two timestamps per run, with distinct meanings​

  • scheduled_for — the canonical, un-jittered planned instant. It is the idempotency anchor (sha256(schedule_id || scheduled_for), ADR 0034), the value in the signature payload and X-Scheduled-For header, the tenant-visible history timestamp, and the partition key (docs/03). Jitter never touches it.
  • fire_after — scheduled_for + deterministic_jitter; the earliest instant the executor may claim the run. Jitter only moves fire_after.

2. Jitter is deterministic, not random​

jitter = hash(schedule_id) mod jitter_seconds. The same occurrence of the same schedule always fires at the same offset — replay, reconciliation, and debugging stay predictable, and the offset is stable across replicas and retries.

3. Defaults that target the herd, not everyone​

Default jitter_seconds = min(60, interval/10) for schedules landing on :00; 0 otherwise. Tenants may set 0 explicitly (exact firing) or up to 900. Jitter is one of five herd controls — alongside the planner's 5 s lookahead, the durable queue (a spike becomes backlog, not loss), backlog-driven autoscaling, and a global outbound token bucket — with the SLO: p99 fire delay ≤ 2 s normally, ≤ 15 s at :00.

Alternatives Considered​

  • Random per-fire jitter. Rejected: two replicas (or a retry after crash) would compute different fire instants for the same occurrence; reconciliation ("why did this fire at 12:00:37?") becomes unanswerable, and test fixtures become flaky by construction.
  • No jitter, absorb the herd with capacity. Rejected: provisioning executor and egress capacity for the :00 spike wastes it the other 59 minutes; deterministic spreading is free and removes the spike at its source. The durable queue still absorbs what jitter does not.
  • Jittering scheduled_for itself. Rejected outright: it would change the idempotency key per computed offset, break the tenant's "my 03:00 job" mental model in history and signatures, and couple correctness (dedup) to a load-shedding concern.
  • Mandatory jitter for all schedules. Rejected: some tenants legitimately need exact firing (market-open jobs); jitter defaults on only where the herd forms (:00 alignments) and stays tenant-overridable to 0.

Consequences​

  • Positive: Herd control is invisible to the tenant's correctness model — keys, signatures, and history all speak scheduled_for; only wall-clock delivery time moves, within a bounded, documented window.
  • Positive: Determinism makes the fire time of any occurrence computable from the schedule row alone — support can answer "when will/did it fire" without logs.
  • Negative / accepted: A :00 schedule with default jitter fires up to 60 s "late" by wall clock. Documented; tenants who care set jitter_seconds: 0.
  • Negative / accepted: Deterministic offsets mean one schedule's offset never changes — a tenant whose jittered slot collides with their own other traffic must adjust jitter_seconds manually rather than benefiting from re-rolls.
  • Interaction: fire_after is what the executor claim index orders on (docs/03 job_run_claim_idx via next_attempt_at); the SLO metric scheduler_fire_delay_seconds measures scheduled_for → first attempt, so jitter is visible in the histogram by design.