The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a job scheduler?
Tons of work shouldn’t happen now — send this reminder at 9am, bill this card monthly, retry that webhook in 30 seconds, run this report nightly. You need something that reliably runs the right job at the right time, even across thousands of machines and through failures.
Split the problem in two: remember when each job should run, and run it when the time comes. Persist schedules durably, scan for what’s due, and hand due jobs to workers. The hard parts are not missing a job and not double-firing one.
What the new pieces do
- Clientclient
- Registers work to run later — once at a time (“email me at 9am”) or on a recurring schedule (“every night”). Then it walks away and trusts the scheduler.
Step 1 · Remember the work
Register & persist jobs
A job submitted now might need to run next month. If the schedule lives only in memory, a single restart forgets it — and a forgotten payroll run is a very bad day.
A job submitted now might need to run next month. Where does the schedule live?
A single restart forgets every in-memory timer — and a forgotten payroll run is a very bad day. Schedules must survive crashes, so memory alone won’t do.
That just moves the problem to the client and assumes it stays alive, online and trustworthy for a month. The whole point is to hand off and walk away.
Persist what/when/how-often before acknowledging, indexed by next run time. Now the system can crash and recover without losing a job, and "what’s due?" is a cheap range read.
A Scheduler API accepts job definitions (what, when, how often) and writes them to a durable Job Store, indexed by next run time. One-off jobs store a single time; recurring jobs store a cron expression and compute the next time after each run.
Why this piece earns its place
The solution says to store the cron expression, but a cron expression on its own is not a rule — it is a rule missing its clock. “Every night” is a local hour, and unless the timezone it was written in is persisted beside it, a daylight-saving shift silently moves every fire by the offset for part of the year, and on the night the clock jumps the local hour the job was waiting for either happens twice or does not exist at all. The payload deserves the same suspicion. A job registered today may not fire until next month, so anything captured by value — a price, an address, a feature flag — is a stale snapshot by the time a worker reads it. Store a reference and resolve it at execution time for anything the business is allowed to change underneath you. The store is holding intent that has to survive time, not a photograph of the moment someone pressed submit.
What the new pieces do
- Scheduler APIbackend
- Accepts job definitions: what to run, when, how often, and with what payload. Validates and persists them; it doesn’t run anything itself.
- Job Storestore
- The durable record of every job, its next run time, and its status. Indexed by next-run-time so “what’s due?” is a cheap range query.
Step 2 · Notice it’s time
Scan for what’s due
Jobs sit in the store with future run times. Something has to keep checking the clock and notice the instant a job becomes due — without scanning the entire table every second.
Jobs sit with future run times. How do you notice the instant one is due, without scanning the whole table every second?
Scanning hundreds of millions of rows every second is enormous wasted I/O. You should touch only the jobs that are actually due, not the entire table.
Because the store is indexed by run time, "what’s due?" is a cheap bounded range read — pull the due ones, leave the rest untouched. A short poll keeps latency low and cost flat.
Millions of live OS timers don’t fit in one process and vanish on restart — back to the durability problem. A polled durable index survives crashes and scales.
A Due Scanner periodically queries the store for jobs whose next_run_time ≤ now. Because the store is indexed by run time, that’s a cheap bounded range read — pull the due ones, leave the rest untouched.
Why this piece earns its place
The poll interval is the knob this step introduces, and it is a floor on lateness before it is anything else: a job can only be as punctual as the gap between scans, so “every second” is really a promise that a job fires within about a second of its time. Shorten it and you pay for range reads that mostly come back empty, all aimed at the same narrow end of one index — the hottest, most contended range in the store, hit by every scanner you run. Lengthen it and everything on the system is uniformly late. The part people forget is the bound on the read. A query for next_run_time ≤ now has no natural size: after an outage, or when a shard falls behind, “due” isn’t the last tick’s worth of work, it is every job that came due while you were down. Without a limit per scan and a stable sort that resumes where the previous batch stopped, the first scan after recovery tries to load the whole backlog in one query and takes the scanner down again, right as it is needed most.
What the new pieces do
- Due Scannerservice
- Periodically asks the store for jobs whose run time has arrived and moves them toward execution. The clock-watcher that turns time into action.
Back of the envelope
- index by next_run_time
- "what’s due?" = a bounded range read
- poll ~every 1s
- low latency, flat cost — touch only due jobs
- future jobs untouched
- cost independent of total job count
Step 3 · Don’t run it inline
Decouple detect from execute
If the scanner also executed jobs, a slow job (or a flood of due jobs at the top of the hour) would stall scanning — and the next batch of due jobs would be late. Detection and execution have completely different scaling needs.
At the top of the hour thousands of jobs come due at once and some run slowly. What stops scanning falling behind?
If the scanner executes inline, one slow job or a burst stalls scanning and the next due batch is late. Detection must stay punctual; execution can lag.
A longer interval makes every job fire late and still stalls when a burst hits. The fix is to separate detecting from doing, not to slow detection.
A Ready Queue between them keeps scanning fast and punctual while workers scale out to absorb bursty execution. Each side scales and fails independently.
The scanner just drops due jobs onto a Ready Queue; a separate Worker Pool drains it and runs them. Now scanning stays fast and punctual while workers scale out to absorb bursty execution load.
Why this piece earns its place
The queue removes the stall and introduces a new problem: the scanner now runs on a timer that is faster than the work. A job sits in the store still marked due while workers are busy, so the next scan a second later reads the same row and enqueues it again, and the scan after that makes a third copy — one due job becomes a flood, and the harder the workers are struggling the faster you pile on. So enqueueing has to change the job’s state in the same breath: the scanner claims the row — flipping its status, or pushing next_run_time out by a claim window — and only then writes to the queue, so the following scan simply doesn’t see it. The queue itself has to be durable for the same reason the store is. An in-memory broker sitting between a persisted schedule and a persisted result quietly turns “we never forget a job” back into “we forget everything in flight”, and a worker outage — exactly when its buffer is deepest — is when you find that out.
- 1sscan interval
- queueabsorbs bursts
- Nworkers
What the new pieces do
- Ready Queuebus
- Holds jobs that are due right now. Decouples deciding when from doing the work, and lets execution scale and retry independently.
- Worker Poolworker
- Pulls due jobs off the queue and actually runs them. A stateless fleet you scale to match how much work comes due at once.
Step 4 · Run it once, really
At-least-once + idempotency
Queues deliver at least once, and a worker can crash after doing the work but before marking it done — so the job runs again. Charging a card twice because of a retry is unacceptable.
A worker crashes after doing the work but before marking it done, so the job is redelivered. How do you not run it twice?
Mark-then-run means a crash after marking but before doing silently skips the job (at-most-once) — a missed payroll run. You can’t avoid double-execution by risking zero-execution.
Each execution carries a stable key; the worker checks the log first and writes the result after, so a redelivered job is recognized and skipped. At-least-once delivery + dedup = effectively-once.
Exactly-once delivery is impossible across worker crashes — the worker may act, then die before acking. Make the execution idempotent instead of trusting the transport.
Give every execution an idempotency key and record outcomes in an Execution Log. A worker checks the log before acting and writes the result after, so a redelivered job is recognized and skipped. Mark the job done in the store and compute its next run.
Why this piece earns its place
All of this rests on what the key is made of, and it is easy to build one that dedupes nothing. A key minted per delivery is unique every time, so every redelivery looks new. A key that is only the job id fails the other way: it collides with tomorrow’s run of the same recurring job, and the second night is skipped as a duplicate. The key has to name the occurrence — job id plus the scheduled fire time, or a run sequence — so every delivery of one firing collides and no two firings ever do. Then there is the gap the log cannot close. Unless the effect and the log entry commit together, a worker can act and die before recording it, which is the case that opened this step. What you can do is write the key before acting and the outcome after, so a crash leaves a row saying attempted rather than nothing, and recovery has something to ask about — replay the key against the downstream that owns the effect instead of guessing. And give the log a retention policy: it grows with every execution forever, and it sits on the hot path of every one of them.
What the new pieces do
- Execution Logstore
- Records each run’s outcome and its idempotency key, so a job that gets delivered twice is executed once — the dedup ledger.
Back of the envelope
- at-least-once delivery
- crash-after-work ⇒ redelivery is normal
- stable key + execution log
- check before, record after ⇒ run once
- effectively-once
- exactly-once delivery is impossible across crashes
Step 5 · Don’t fire five times
Leader election
For availability you run several scheduler instances. But if all of them scan and enqueue the same due job, it fires once per instance — the duplicate problem moved upstream of the workers.
You run several scheduler instances for availability. How do you stop all of them firing the same due job?
Relying on downstream dedup for every job wastes work and leans entirely on idempotency being perfect. Better not to enqueue the same job N times in the first place.
A single instance is a single point of failure — when it dies, nothing fires until someone restarts it. You need multiple instances for availability without multiple fires.
Only the leaseholder scans-and-enqueues a partition; standbys take over instantly if its lease expires. Coordinating who-scans is far cheaper than coordinating every job — single-firing plus fast failover.
Use leader election (via a lease in a consensus store / distributed lock) so only one instance owns scanning for a given partition of jobs at a time. Others stand by, ready to take over instantly if the leader’s lease expires.
Why this piece earns its place
A lease buys failover; it does not buy mutual exclusion, and that is worth saying out loud because the diagram makes it look airtight. A leader can stop for a long garbage-collection pause, lose the network for a moment, or be descheduled. Its lease expires, a standby is elected and starts scanning, and then the original wakes up with no idea any time passed — and its very next act is to enqueue the batch it was holding. For that window two instances both believe they own the partition. A shorter lease doesn’t close the gap; it only makes it more frequent. What you can do is make the collision detectable: the leaseholder carries a fencing token that increases with every election and stamps it on each claim it writes, and the store refuses any write carrying a token older than one it has already accepted, so the woken leader’s work is rejected instead of doubled. Lease duration then becomes a plain availability knob — short means fast takeover and elections triggered by ordinary pauses, long means nothing in that partition fires until it expires. This is also why idempotency still matters underneath: the coordinator reduces double-fires, it cannot abolish them.
What the new pieces do
- Coordinatorservice
- Ensures only one scheduler instance scans-and-enqueues a given partition, so a due job isn’t fired five times by five running schedulers.
Step 6 · When jobs fail
Retries, backoff & DLQ
Jobs call flaky downstreams that time out or error. Retrying instantly and forever just hammers a struggling service and clogs the workers with a job that will never succeed.
A job calls a flaky downstream that errors. How do you retry without hammering it or looping forever?
Instant infinite retries hammer the struggling downstream and clog workers with a job that may never succeed. Retries need spacing and a limit.
Re-queue with growing delays (protecting the downstream), cap attempts, and send exhausted jobs to a dead-letter queue for humans — instead of one poison job looping forever and clogging the pipeline.
Dropping on first failure loses work that was a transient blip away from succeeding. Transient failures deserve bounded retries; only persistent ones get parked in the DLQ.
On failure, re-queue with exponential backoff (and jitter) up to a max attempts. Jobs that exhaust their retries move to a dead-letter queue for inspection instead of looping. Successful retries record normally in the execution log.
Why this piece earns its place
Backoff assumes the failure is worth repeating, and most retry budgets get burned on failures that never were. A timeout or a refused connection is a blip; a malformed payload, a deleted account, a permission that was revoked will fail identically on every attempt — and because the delays grow, the last attempts are the longest, so a job that was hopeless from the first try occupies the pipeline longest before anyone is told about it. Classify the failure where it is caught: retry the transport-shaped ones, dead-letter the argument-shaped ones immediately. The other half is that a DLQ is where the design quietly stops being automatic. Nothing drains it, so without an alarm on its depth it is a silent drop with better paperwork — and the jobs in it are precisely the ones somebody wanted. Give it an owner, an alert and a replay path, and make replay re-enter through the original occurrence’s key, because a timeout is ambiguous: the job that “failed” may have already done its work, and a replayed payment that quietly succeeded the first time is the exact outcome this whole design exists to prevent.
What the new pieces do
- Retries + DLQstore
- Failed jobs are re-queued with exponential backoff; ones that keep failing land in a dead-letter queue for humans to inspect, not retried forever.
Back of the envelope
- delay = base × 2^attempt + jitter
- protects the flaky downstream
- cap at max attempts
- then dead-letter — don’t loop forever
- DLQ ⇒ a human inspects
- one poison job can’t clog the pipeline
Step 7 · Scale & time
The sharp edges
Polling a store with hundreds of millions of jobs every second is wasteful, and machine clock skew means two nodes disagree on “now”. At scale, a single scanner is also a bottleneck.
Replace second-by-second polling with a hierarchical time wheel for efficient near-term scheduling, and shard jobs (by id or tenant) so many leader-elected scanners run in parallel. Treat run times as approximate and lean on idempotency to absorb clock skew.
Why this piece earns its place
A time wheel is an in-memory structure — the thing step 1 refused to trust — so it only works if it is treated as a cache of the near future rather than the schedule itself. The store stays the record; the wheel holds a bounded horizon of imminent jobs, and a slower sweep promotes rows into it as that horizon slides forward. The horizon is the knob. Keep it short and the sweep runs almost constantly, which is polling again with extra steps; stretch it and the wheel holds a large slice of the schedule in memory, which costs memory and makes it a bigger thing to rebuild. Rebuild is the part to say out loud, because it happens far more often than a crash: every failover hands a partition to a standby whose wheel is empty, so a new leader’s first act is reloading its horizon from the store, and takeover is only as fast as that read. And a job edited after it was promoted now exists in two places — the wheel has to be invalidated on write, or you cheerfully fire the version of the job the user already cancelled.
You did it
You just designed a job scheduler.
Everything you assembled, in order
- A Scheduler API persists jobs to a store indexed by next-run-time.
- A due scanner polls that time index for jobs that have come due.
- A ready queue decouples due-detection from a scalable worker pool.
- Idempotency keys + an execution log make at-least-once delivery effectively-once.
- Leader election per partition stops multiple schedulers double-firing a job.
- Exponential backoff and a dead-letter queue handle failures without looping.
- Time wheels and sharding scale scanning; idempotency absorbs clock skew.