The big idea
What is a job scheduler?
Tons of work shouldn’t happen now — send this reminder at 9am, bill this card monthly, retry that webhook in 30 seconds, run this report nightly. You need something that reliably runs the right job at the right time, even across thousands of machines and through failures.
Split the problem in two: remember when each job should run, and run it when the time comes. Persist schedules durably, scan for what’s due, and hand due jobs to workers. The hard parts are not missing a job and not double-firing one.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · Remember the work
Register & persist jobs
A job submitted now might need to run next month. If the schedule lives only in memory, a single restart forgets it — and a forgotten payroll run is a very bad day.
Design decision: A job submitted now might need to run next month. Where does the schedule live?
The call: In a durable Job Store indexed by next-run-time. — Persist what/when/how-often before acknowledging, indexed by next run time. Now the system can crash and recover without losing a job, and "what’s due?" is a cheap range read.
A Scheduler API accepts job definitions (what, when, how often) and writes them to a durable Job Store, indexed by next run time. One-off jobs store a single time; recurring jobs store a cron expression and compute the next time after each run.
Durability first: The schedule is the asset. Persist it before acknowledging, index it by next-run-time, and the rest of the system can crash and recover without losing a single job.
Step 2 · Notice it’s time
Scan for what’s due
Jobs sit in the store with future run times. Something has to keep checking the clock and notice the instant a job becomes due — without scanning the entire table every second.
Design decision: Jobs sit with future run times. How do you notice the instant one is due, without scanning the whole table every second?
The call: Range-query the time-sorted index for next_run_time ≤ now. — Because the store is indexed by run time, "what’s due?" is a cheap bounded range read — pull the due ones, leave the rest untouched. A short poll keeps latency low and cost flat.
A Due Scanner periodically queries the store for jobs whose next_run_time ≤ now. Because the store is indexed by run time, that’s a cheap bounded range read — pull the due ones, leave the rest untouched.
Poll the index: A short poll interval (say, every second) over a time-sorted index keeps latency low and cost flat. We’ll upgrade this to a time wheel later for huge job counts.
Step 3 · Don’t run it inline
Decouple detect from execute
If the scanner also executed jobs, a slow job (or a flood of due jobs at the top of the hour) would stall scanning — and the next batch of due jobs would be late. Detection and execution have completely different scaling needs.
Design decision: At the top of the hour thousands of jobs come due at once and some run slowly. What stops scanning falling behind?
The call: Scanner enqueues due jobs; a worker pool drains and runs them. — A Ready Queue between them keeps scanning fast and punctual while workers scale out to absorb bursty execution. Each side scales and fails independently.
The scanner just drops due jobs onto a Ready Queue; a separate Worker Pool drains it and runs them. Now scanning stays fast and punctual while workers scale out to absorb bursty execution load.
Separate the clock from the muscle: Detecting “it’s time” must stay real-time; doing the work can lag and parallelize. A queue between them lets each scale and fail independently — the recurring theme of these designs.
Step 4 · Run it once, really
At-least-once + idempotency
Queues deliver at least once, and a worker can crash after doing the work but before marking it done — so the job runs again. Charging a card twice because of a retry is unacceptable.
Design decision: A worker crashes after doing the work but before marking it done, so the job is redelivered. How do you not run it twice?
The call: Idempotency key + execution log: check before acting, record after. — Each execution carries a stable key; the worker checks the log first and writes the result after, so a redelivered job is recognized and skipped. At-least-once delivery + dedup = effectively-once.
Give every execution an idempotency key and record outcomes in an Execution Log. A worker checks the log before acting and writes the result after, so a redelivered job is recognized and skipped. Mark the job done in the store and compute its next run.
Effectively-once: True exactly-once delivery is impossible across failures, so make execution idempotent instead: at-least-once delivery + dedup on a stable key = the job’s effect happens once.
Step 5 · Don’t fire five times
Leader election
For availability you run several scheduler instances. But if all of them scan and enqueue the same due job, it fires once per instance — the duplicate problem moved upstream of the workers.
Design decision: You run several scheduler instances for availability. How do you stop all of them firing the same due job?
The call: Leader election: one instance owns scanning a partition via a lease. — Only the leaseholder scans-and-enqueues a partition; standbys take over instantly if its lease expires. Coordinating who-scans is far cheaper than coordinating every job — single-firing plus fast failover.
Use leader election (via a lease in a consensus store / distributed lock) so only one instance owns scanning for a given partition of jobs at a time. Others stand by, ready to take over instantly if the leader’s lease expires.
Single writer per partition: Coordinating who scans is far cheaper than coordinating every job. A short-lived lease gives you single-firing during normal operation and fast failover when a leader dies.
Step 6 · When jobs fail
Retries, backoff & DLQ
Jobs call flaky downstreams that time out or error. Retrying instantly and forever just hammers a struggling service and clogs the workers with a job that will never succeed.
Design decision: A job calls a flaky downstream that errors. How do you retry without hammering it or looping forever?
The call: Exponential backoff with jitter, max attempts, then a DLQ. — Re-queue with growing delays (protecting the downstream), cap attempts, and send exhausted jobs to a dead-letter queue for humans — instead of one poison job looping forever and clogging the pipeline.
On failure, re-queue with exponential backoff (and jitter) up to a max attempts. Jobs that exhaust their retries move to a dead-letter queue for inspection instead of looping. Successful retries record normally in the execution log.
Fail gracefully, then give up: Backoff protects the downstream and the scheduler; a DLQ keeps a single poison job from clogging the pipeline. Retrying forever is how one bad job takes everything down.
Step 7 · Scale & time
The sharp edges
Polling a store with hundreds of millions of jobs every second is wasteful, and machine clock skew means two nodes disagree on “now”. At scale, a single scanner is also a bottleneck.
Replace second-by-second polling with a hierarchical time wheel for efficient near-term scheduling, and shard jobs (by id or tenant) so many leader-elected scanners run in parallel. Treat run times as approximate and lean on idempotency to absorb clock skew.
Time wheels & sharding: A time wheel buckets jobs by when they fire, so you touch only the imminent ones. Sharding spreads both storage and scanning. Never assume perfectly synced clocks — design for “about now”.
You did it
You just designed a job scheduler.
- A Scheduler API persists jobs to a store indexed by next-run-time.
- A due scanner polls that time index for jobs that have come due.
- A ready queue decouples due-detection from a scalable worker pool.
- Idempotency keys + an execution log make at-least-once delivery effectively-once.
- Leader election per partition stops multiple schedulers double-firing a job.
- Exponential backoff and a dead-letter queue handle failures without looping.
- Time wheels and sharding scale scanning; idempotency absorbs clock skew.