The whole design, in writing
Learn system design by building a distributed job scheduler / cron step by step. An interactive guide covering the job store, due-time scanning, decoupling detection from execution, at-least-once execution with idempotency, leader election to avoid double-fires, retries with backoff and DLQ, and scaling.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a job scheduler?
Tons of work shouldn’t happen now — send this reminder at 9am, bill this card monthly, retry that webhook in 30 seconds, run this report nightly. You need something that reliably runs the right job at the right time, even across thousands of machines and through failures.
Split the problem in two: remember when each job should run, and run it when the time comes. Persist schedules durably, scan for what’s due, and hand due jobs to workers. The hard parts are not missing a job and not double-firing one.
What the new pieces do
- Clientclient
- Registers work to run later — once at a time (“email me at 9am”) or on a recurring schedule (“every night”). Then it walks away and trusts the scheduler.
Step 1 · Remember the work
Register & persist jobs
A job submitted now might need to run next month. If the schedule lives only in memory, a single restart forgets it — and a forgotten payroll run is a very bad day.
A job submitted now might need to run next month. Where does the schedule live?
A single restart forgets every in-memory timer — and a forgotten payroll run is a very bad day. Schedules must survive crashes, so memory alone won’t do.
That just moves the problem to the client and assumes it stays alive, online and trustworthy for a month. The whole point is to hand off and walk away.
Persist what/when/how-often before acknowledging, indexed by next run time. Now the system can crash and recover without losing a job, and "what’s due?" is a cheap range read.
A Scheduler API accepts job definitions (what, when, how often) and writes them to a durable Job Store, indexed by next run time. One-off jobs store a single time; recurring jobs store a cron expression and compute the next time after each run.
What the new pieces do
- Scheduler APIbackend
- Accepts job definitions: what to run, when, how often, and with what payload. Validates and persists them; it doesn’t run anything itself.
- Job Storestore
- The durable record of every job, its next run time, and its status. Indexed by next-run-time so “what’s due?” is a cheap range query.
Step 2 · Notice it’s time
Scan for what’s due
Jobs sit in the store with future run times. Something has to keep checking the clock and notice the instant a job becomes due — without scanning the entire table every second.
Jobs sit with future run times. How do you notice the instant one is due, without scanning the whole table every second?
Scanning hundreds of millions of rows every second is enormous wasted I/O. You should touch only the jobs that are actually due, not the entire table.
Because the store is indexed by run time, "what’s due?" is a cheap bounded range read — pull the due ones, leave the rest untouched. A short poll keeps latency low and cost flat.
Millions of live OS timers don’t fit in one process and vanish on restart — back to the durability problem. A polled durable index survives crashes and scales.
A Due Scanner periodically queries the store for jobs whose next_run_time ≤ now. Because the store is indexed by run time, that’s a cheap bounded range read — pull the due ones, leave the rest untouched.
What the new pieces do
- Due Scannerservice
- Periodically asks the store for jobs whose run time has arrived and moves them toward execution. The clock-watcher that turns time into action.
Back of the envelope
- index by next_run_time
- "what’s due?" = a bounded range read
- poll ~every 1s
- low latency, flat cost — touch only due jobs
- future jobs untouched
- cost independent of total job count
Step 3 · Don’t run it inline
Decouple detect from execute
If the scanner also executed jobs, a slow job (or a flood of due jobs at the top of the hour) would stall scanning — and the next batch of due jobs would be late. Detection and execution have completely different scaling needs.
At the top of the hour thousands of jobs come due at once and some run slowly. What stops scanning falling behind?
If the scanner executes inline, one slow job or a burst stalls scanning and the next due batch is late. Detection must stay punctual; execution can lag.
A longer interval makes every job fire late and still stalls when a burst hits. The fix is to separate detecting from doing, not to slow detection.
A Ready Queue between them keeps scanning fast and punctual while workers scale out to absorb bursty execution. Each side scales and fails independently.
The scanner just drops due jobs onto a Ready Queue; a separate Worker Pool drains it and runs them. Now scanning stays fast and punctual while workers scale out to absorb bursty execution load.
- 1sscan interval
- queueabsorbs bursts
- Nworkers
What the new pieces do
- Ready Queuebus
- Holds jobs that are due right now. Decouples deciding when from doing the work, and lets execution scale and retry independently.
- Worker Poolworker
- Pulls due jobs off the queue and actually runs them. A stateless fleet you scale to match how much work comes due at once.
Step 4 · Run it once, really
At-least-once + idempotency
Queues deliver at least once, and a worker can crash after doing the work but before marking it done — so the job runs again. Charging a card twice because of a retry is unacceptable.
A worker crashes after doing the work but before marking it done, so the job is redelivered. How do you not run it twice?
Mark-then-run means a crash after marking but before doing silently skips the job (at-most-once) — a missed payroll run. You can’t avoid double-execution by risking zero-execution.
Each execution carries a stable key; the worker checks the log first and writes the result after, so a redelivered job is recognized and skipped. At-least-once delivery + dedup = effectively-once.
Exactly-once delivery is impossible across worker crashes — the worker may act, then die before acking. Make the execution idempotent instead of trusting the transport.
Give every execution an idempotency key and record outcomes in an Execution Log. A worker checks the log before acting and writes the result after, so a redelivered job is recognized and skipped. Mark the job done in the store and compute its next run.
What the new pieces do
- Execution Logstore
- Records each run’s outcome and its idempotency key, so a job that gets delivered twice is executed once — the dedup ledger.
Back of the envelope
- at-least-once delivery
- crash-after-work ⇒ redelivery is normal
- stable key + execution log
- check before, record after ⇒ run once
- effectively-once
- exactly-once delivery is impossible across crashes
Step 5 · Don’t fire five times
Leader election
For availability you run several scheduler instances. But if all of them scan and enqueue the same due job, it fires once per instance — the duplicate problem moved upstream of the workers.
You run several scheduler instances for availability. How do you stop all of them firing the same due job?
Relying on downstream dedup for every job wastes work and leans entirely on idempotency being perfect. Better not to enqueue the same job N times in the first place.
A single instance is a single point of failure — when it dies, nothing fires until someone restarts it. You need multiple instances for availability without multiple fires.
Only the leaseholder scans-and-enqueues a partition; standbys take over instantly if its lease expires. Coordinating who-scans is far cheaper than coordinating every job — single-firing plus fast failover.
Use leader election (via a lease in a consensus store / distributed lock) so only one instance owns scanning for a given partition of jobs at a time. Others stand by, ready to take over instantly if the leader’s lease expires.
What the new pieces do
- Coordinatorservice
- Ensures only one scheduler instance scans-and-enqueues a given partition, so a due job isn’t fired five times by five running schedulers.
Step 6 · When jobs fail
Retries, backoff & DLQ
Jobs call flaky downstreams that time out or error. Retrying instantly and forever just hammers a struggling service and clogs the workers with a job that will never succeed.
A job calls a flaky downstream that errors. How do you retry without hammering it or looping forever?
Instant infinite retries hammer the struggling downstream and clog workers with a job that may never succeed. Retries need spacing and a limit.
Re-queue with growing delays (protecting the downstream), cap attempts, and send exhausted jobs to a dead-letter queue for humans — instead of one poison job looping forever and clogging the pipeline.
Dropping on first failure loses work that was a transient blip away from succeeding. Transient failures deserve bounded retries; only persistent ones get parked in the DLQ.
On failure, re-queue with exponential backoff (and jitter) up to a max attempts. Jobs that exhaust their retries move to a dead-letter queue for inspection instead of looping. Successful retries record normally in the execution log.
What the new pieces do
- Retries + DLQstore
- Failed jobs are re-queued with exponential backoff; ones that keep failing land in a dead-letter queue for humans to inspect, not retried forever.
Back of the envelope
- delay = base × 2^attempt + jitter
- protects the flaky downstream
- cap at max attempts
- then dead-letter — don’t loop forever
- DLQ ⇒ a human inspects
- one poison job can’t clog the pipeline
Step 7 · Scale & time
The sharp edges
Polling a store with hundreds of millions of jobs every second is wasteful, and machine clock skew means two nodes disagree on “now”. At scale, a single scanner is also a bottleneck.
Replace second-by-second polling with a hierarchical time wheel for efficient near-term scheduling, and shard jobs (by id or tenant) so many leader-elected scanners run in parallel. Treat run times as approximate and lean on idempotency to absorb clock skew.
You did it
You just designed a job scheduler.
Everything you assembled, in order
- A Scheduler API persists jobs to a store indexed by next-run-time.
- A due scanner polls that time index for jobs that have come due.
- A ready queue decouples due-detection from a scalable worker pool.
- Idempotency keys + an execution log make at-least-once delivery effectively-once.
- Leader election per partition stops multiple schedulers double-firing a job.
- Exponential backoff and a dead-letter queue handle failures without looping.
- Time wheels and sharding scale scanning; idempotency absorbs clock skew.