ADR 0034 — Scheduler Execution Semantics: At-Least-Once Delivery, Exactly-Once Run Materialisation, Tenant-Facing Idempotency Key

Status: Accepted Context date: 2026-08-06 Builds on: ADR 0033 (claim-queue scheduling)

Context

No distributed scheduler can promise exactly-once delivery over HTTP: a 2xx acknowledgment lost on the wire is indistinguishable from a failed attempt, so the deliverer must either redeliver (at-least-once) or accept silent loss (at-most-once). This choice is the scheduler's public semantic contract — tenants build on it, and changing it later is breaking. It had to be decided, stated precisely, and given tooling.

Decision

The semantics, layer by layer, are explicit:

Layer	Semantic	Mechanism
Fire detection	at-least-once	A crashed executor's lease expires; the run is reclaimed and re-attempted.
Run materialisation	exactly-once	UNIQUE (schedule_id, scheduled_for) — one durable job_run row per planned occurrence, regardless of replica races or clock jumps (ADR 0033).
Delivery to the tenant	at-least-once	A 2xx lost on the wire causes redelivery; retries and lease-reclaims may repeat a POST.
Tenant-side effect	exactly-once via tooling, not promise	Every request carries a stable idempotency key; the documented contract is dedupe on the key.

Every delivery request carries:

• X-Idempotency-Key: sha256(schedule_id || scheduled_for) — stable across all attempts and all replicas of one run, because it derives from the canonical un-jittered scheduled_for (ADR 0037), never from attempt state.
• X-Schedule-Id, X-Run-Id, X-Attempt, X-Scheduled-For, and the HMAC signature (X-Signature, ADR 0032's shared scheme).

The tenant documentation states the contract plainly: you may receive a request more than once; deduplicate on X-Idempotency-Key.

Alternatives Considered

• At-most-once delivery. Rejected: a silently missed billing rollup at 03:00 is a support escalation with nothing to point at; a duplicate the tenant can dedupe is documented behaviour. When forced to choose, a scheduler product must choose "never silently lost".
• Promising exactly-once delivery. Rejected as physically impossible over HTTP (lost-ack problem). Promising it and violating it under failure is worse than documenting at-least-once.
• Per-attempt idempotency keys. Rejected: the key would no longer identify the occurrence, so tenants could not dedupe the exact duplicates the semantics permit — it would defeat the tooling's purpose.
• Exactly-once effect via a tenant-side handshake (two-phase confirm). Rejected: doubles the integration surface for every tenant to solve a problem a single header solves for the common case.

Consequences

• Positive: The invariants are enforced by the database (unique index), not by clocks or coordination — backwards clock jumps and replica races cannot double-materialise a run.
• Positive: One run has one key across every retry, executor crash, and lease reclaim — tenant-side dedup is a set-membership check.
• Negative / accepted: A tenant may observe a duplicate POST with the same idempotency key after an executor crash mid-delivery. This is documented, expected behaviour — exactly why the key exists.
• Negative / accepted: schedule.run.succeeded reflects "we saw a 2xx", not "the tenant's effect happened exactly once" — the effect guarantee belongs to the tenant's dedup.
• Documentation duty: The at-least-once contract and the dedupe instruction must appear in the tenant-facing API docs (contracts/openapi/scheduler.yaml webhook callback description) and SDK guides; the semantic is frozen by this ADR.