Skip to main content

ADR 0041 — Scheduler Single-Region Planning and DR on Regional Failover

Status: Accepted Context date: 2026-08-06 Amends: ADR 0001 (multi-region strategy) — adds the scheduler's plane assignment Builds on: ADR 0033 (claim-queue), ADR 0036 (misfire policy)

Context​

ADR 0001 splits the platform by plane: media is region-local and room-pinned; control-plane data is single-writer per tenant home region with async cross-region replicas. The scheduler is a new kind of workload — it originates actions on a clock rather than responding to requests — so its regional assignment had to be decided explicitly: where does planning run, where do outbound deliveries originate, and what exactly happens when a region is lost?

Decision​

1. The scheduler is control-plane: single-writer, tenant home region​

A tenant's schedules and runs live in the tenant's home-region scheduler_db (one primary, async cross-region replicas — the ADR 0001 posture). Planner and executor replicas run in that region only; outbound webhook deliveries originate from the home region's egress. There is no cross-region planner coordination in normal operation — the ADR 0033 guards (SKIP LOCKED + unique-occurrence index) operate within one primary.

2. DR is regional failover with a cold-start planner​

On home-region loss, the runbook promotes the replica (RPO seconds, per ADR 0001) and the standby region's scheduler deployment starts cold: it reads durable next_fire_at state and applies each schedule's misfire policy (ADR 0036) to the outage window. This is precisely why fire_once is the platform default — a fleet-wide cold start fires each affected schedule at most once, bounded and useful, instead of flooding or silently skipping.

Two consequences of async replication are accepted and folded into the existing semantics:

  • A run that was materialised or delivered just before failover may be absent from the promoted replica; on replan it can fire again. This degrades to at-least-once across regions — exactly the delivery contract tenants already hold (ADR 0034), with the same stable idempotency key (derived from schedule_id + scheduled_for, not from the lost row).
  • Runs recorded as terminal on the lost primary but not replicated reappear as pending and are re-attempted; the tenant's dedup absorbs them.

3. Out of scope, with the seam named​

Cross-region active-active planning (two planners sharing one schedule set) is explicitly out of scope. If a future requirement forces it, the seam is the same one ADR 0033 names for scale-out: run materialisation stays anchored to one writable store per tenant; only claim distribution changes. Cross-region within one tenant would additionally require a globally consistent unique-occurrence guard — a real distributed-systems cost this ADR declines to pay before it is needed.

Alternatives Considered​

  • Active-active planners in two regions. Rejected: the unique-occurrence index is the correctness backbone (ADR 0033), and it only holds within one primary. Active-active either double-fires or requires consensus/global uniqueness — heavy machinery to shave minutes off a rare failover that at-least-once semantics already cover.
  • A global scheduler region for all tenants. Rejected: it breaks ADR 0001's tenant-home-region data model, concentrates all tenants' blast radius in one region, and puts every delivery's egress far from the tenant's endpoints.
  • Warm standby planner running read-only against the replica. Rejected: it adds a second live component to keep honest for marginal RTO gain; cold start against durable next_fire_at is simple and its behaviour is exactly the already-specified misfire path.
  • Pausing all schedules on failover pending operator review. Rejected: turns every regional incident into a tenant-visible outage of the product; the misfire policies exist precisely to make automated recovery safe per schedule.

Consequences​

  • Positive: The scheduler adds zero new multi-region machinery — it inherits ADR 0001's replication and runbook, and its failure behaviour is composed entirely of already-decided semantics (misfire policy + at-least-once + idempotency key).
  • Positive: Failover behaviour is testable in one region: kill the planner, advance the clock, verify the misfire outcomes — the DR path is the outage path.
  • Negative / accepted: RTO includes replica promotion plus planner cold start; schedules due during that window are handled by misfire policy (default: fire once, late) rather than firing on time.
  • Negative / accepted: Around a failover, tenants may see duplicate deliveries (same idempotency key) and, for skip-policy schedules, missed occurrences recorded per policy. Both are the documented contract, not incident-specific surprises.
  • Amendment: ADR 0001's plane table now reads: media — region-local; control-plane writes — tenant home region; scheduled work — tenant home region, misfire-policy recovery on failover.