Skip to main content

ADR 0036 — Misfire and Catch-Up Policy; Suspension Windows Are Never Backfilled

Status: Accepted Context date: 2026-08-06 Builds on: ADR 0025 (tenant suspension), ADR 0033 (claim-queue scheduling), ADR 0041 (DR behaviour)

Context​

A planner outage, database failover, long deploy, or tenant suspension leaves next_fire_at in the past. What happens next is tenant-visible product behaviour: firing everything that was missed can flood a tenant endpoint with thousands of stale calls; firing nothing silently drops work. The policy has to be explicit, per-schedule, and bounded — and it interacts directly with ADR 0025's suspension machinery.

Decision​

1. Per-schedule misfire_policy, applied when next_fire_at is more than one interval in the past​

PolicyBehaviour
fire_once (default)Materialise only the most recent missed occurrence, then realign to the next future one.
skipRealign to the next future occurrence; nothing fires.
catch_upBackfill missed occurrences, hard-capped at 100 runs / 24 h of history, rate-limited into the executor.

fire_once is the default because it matches DR reality: a standby planner starting cold after a regional failover (ADR 0041) applies the misfire policy to every schedule at once — "run the most recent missed occurrence, once" is the only default that is both safe and useful at fleet scale.

2. Staleness cut-off​

Runs whose scheduled_for is older than max_staleness (default 1 h, tenant-configurable up to 24 h) are marked skipped(reason=stale) instead of being delivered. A 03:00 report delivered at 14:00 is usually worse than no delivery plus a visible skip record.

3. A suspension window is never backfilled — regardless of policy​

On tenant.suspended (ADR 0025 projection), the planner stops materialising and the executor marks claimed pending runs skipped(reason=tenant_suspended); in-flight deliveries finish; schedules are preserved, not deleted. On tenant.activated, schedules resume with the misfire treated as skip even if the schedule's policy is catch_up. A 3-day suspension of a per-minute schedule would otherwise mean 4,320 webhooks on reactivation. An aggregated schedule.run.skipped is emitted so the gap is auditable in run history.

Alternatives Considered​

  • catch_up as the default. Rejected: the default must be safe under fleet-wide events (deploys, failovers); a default that can flood every tenant endpoint simultaneously fails that test. Tenants who genuinely need backfill (e.g. sequential data pipelines) opt in and get the cap.
  • skip as the default. Rejected: for the common "nightly job" case, silently losing the one missed occurrence after a 10-minute outage is exactly the silent-loss failure ADR 0034 rules out. fire_once recovers the most recent intent without flooding.
  • Uncapped catch_up. Rejected: an unbounded backfill after a long outage is a self-inflicted thundering herd; 100 runs / 24 h covers realistic pipelines and turns pathological cases into visible skips.
  • Backfilling suspension windows for catch_up schedules. Rejected: suspension is an administrative action (ADR 0025); replaying the punished period's traffic on reactivation surprises the tenant, hammers their endpoint, and re-litigates the suspension. The gap is recorded, not replayed.
  • Deleting or pausing-forever schedules on suspension. Rejected: suspension is reversible by design (ADR 0025 preserves tenant data); schedules resume like everything else.

Consequences​

  • Positive: Every recovery path — crash, failover, deploy, suspension — has a bounded, pre-decided outcome per schedule, visible in run history as real rows (skipped with a reason), never as silence.
  • Positive: The DR story (ADR 0041) needs no special casing: cold-start planning is the misfire path.
  • Negative / accepted: fire_once means a schedule that missed N occurrences runs once, not N times — tenants with strict per-occurrence pipelines must opt into catch_up and design for its cap.
  • Negative / accepted: The suspension rule overrides tenant configuration (catch_up is ignored for the window). Deliberate: platform safety over per-schedule preference, and auditable via the aggregated skip event.
  • Interaction: Skip reasons (overlap, tenant_suspended, quota_exhausted, stale) are part of the event contract (ADR 0038) and the run-history API — they are the tenant's window into every occurrence that did not fire.