Design Distributed Job Scheduler

Picture an internal platform that other teams call when they need something to happen later. The payments team wants every subscription renewal attempted at the exact minute it falls due. Marketing wants a reminder push sent at 8:00 local time in each of forty timezones. Data engineering wants a nightly export kicked off at 01:30, and a support tool wants to re-open snoozed tickets after three days. None of these teams want to run their own timer process, and none of them want to think about what happens when it crashes.

So the product is a trigger service. A caller registers a job with a schedule and a target, and we promise to deliver a trigger to that target at the right time, retry when it fails, and keep a history of every attempt. The work itself happens in the caller's systems: we call an HTTPS endpoint or publish a message onto a topic. Keeping execution out of scope is a deliberate choice, because it turns a general compute platform into a much smaller timing and delivery problem.

Assume 4 million active jobs owned by a few thousand internal services. About 60% are recurring, mostly hourly or daily, and the rest are one-shot reminders and delayed actions. Assume each trigger payload averages 1.5 KB and each history row about 400 bytes.

Two resources carry the model. A job is the long-lived definition. A run is one scheduled occurrence of it, with its own attempts. Every mutating call takes an Idempotency-Key so a client retry never creates a second job.

Think of the system as three loops that run at different speeds. The registration loop is ordinary CRUD: the API validates a schedule, computes the next fire time in the job's timezone, writes the job row, and inserts one entry into the due index. Nothing on this path waits for the scheduler.

The ticking loop is where timing lives. Jobs are hashed into 1,024 logical shards. A coordinator backed by etcd hands out shards to ticker processes as leases, so each shard has exactly one owner at a time and a crashed ticker's shards move to a survivor within a few seconds. Every second, a ticker reads the entries in its shards that fall due in the next few seconds, claims them with a conditional state change, and publishes a small trigger record to Kafka. Because tickers only read and publish, one process can comfortably run a hundred shards.

The delivery loop is the dispatcher pool. Dispatchers consume the ready topic, make the HTTP call or publish to the target topic, write the attempt to history, and either mark the run finished or schedule a retry. For recurring jobs, the dispatcher that finishes a run, or the ticker that claims it, computes the following occurrence and inserts a fresh index entry. Dispatchers are the only part that scales with target slowness, so they autoscale on in-flight request count, not on CPU.

A plain WHERE next_fire_at <= now() query on the job table works until the table is busy with writes and the planner starts picking bad paths at the top of the hour. Instead, the due index is bucketed by shard and by minute. A ticker that owns shard 173 reads bucket (173, 2026-10-01T09:00) about ten seconds before that minute starts, loads the entries into a small in-memory timing wheel, and releases each one when its second arrives. One indexed range read per shard per minute replaces thousands of polling queries.

Late writes still work. A job created at 09:00:20 that is due at 09:00:40 lands in the current bucket after the ticker has read it, so the ticker also re-reads the current bucket every few seconds, picking up only rows with an inserted_at later than its last read. For one-shot jobs due less than five seconds out, the API can skip the index and publish straight to the ready topic.

Shard leases come from etcd with a 10-second TTL that the ticker renews every 3 seconds. Each time a shard changes hands, its epoch number goes up. The ticker attaches its epoch to every claim, and the claim is a conditional update in the database that succeeds only if the shard's current epoch still matches. This matters because a lease is only a belief. A ticker that froze for 15 seconds still thinks it owns shard 173 when it wakes up, but its epoch is now stale and every claim it attempts updates zero rows. The database, not the ticker's clock, decides.

The claim itself is idempotent: an index entry moves from pending to claimed once. If the ticker crashes between claiming and publishing to Kafka, the entry sits in claimed with no trigger anywhere. A sweeper running on the new owner looks for entries claimed more than 30 seconds ago with no matching run record and republishes them. Using a transactional outbox, where the claim and the outgoing trigger row commit together and a relay ships the row to Kafka, closes that gap completely at the cost of one more table.

Dispatchers commit their Kafka offset only after writing the attempt outcome. If one dies mid-request, another dispatcher receives the same trigger and calls the target again. That is the residual duplicate, and it is unavoidable without the target's help. We shrink it in three ways: the run ID header lets targets drop repeats, a short-lived Redis key SET run_id NX EX 3600 stops two of our own dispatchers from both sending, and webhook timeouts are kept well below the Kafka session timeout so a slow call does not trigger a rebalance and a second delivery.

Retries go to delay tiers rather than sleeping in a dispatcher. There are topics for 5 seconds, 1 minute, 10 minutes and 1 hour. A failed attempt is published to the smallest tier that covers its backoff, and a consumer on that tier holds messages until they are due. After the last attempt, the run is marked dead and the owning team gets an alert, never a silent drop.

Store the cron expression and the timezone, never a pre-computed UTC offset. Compute each next occurrence with a proper tz database at the moment you insert it. On the spring-forward night, a 02:30 job in a zone that skips from 02:00 to 03:00 has no valid local time, so pick a documented rule, usually "fire at 03:00". On the fall-back night, 01:30 happens twice; fire on the first occurrence only. Interviewers like this detail because it is where naive designs double-fire or skip a day.