The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
GPUs are not fungible, interchangeable slots
A generic job scheduler assumes any worker can run any job. GPU scheduling breaks that assumption in several specific ways: distributed training needs MANY GPUs allocated at once, not incrementally; which specific GPUs matter (interconnect distance affects real performance); and GPUs are so expensive that fair allocation across competing teams is a first-class requirement, not an afterthought.
We'll build the scheduler around these specific constraints: all-or-nothing gang scheduling, topology-aware placement, priority-based preemption with checkpointing, and fair-share quotas so scarce capacity doesn't silently concentrate in one team's hands.
What the new pieces do
- Userclient
- A researcher or engineer submitting a training or inference job, competing for a shared pool of genuinely scarce, expensive GPUs.
Step 1 · The baseline
FIFO, first free GPU
Simplest scheduler: jobs are served in the order they arrive, each grabbing whatever GPU is currently free. What breaks specifically for GPU workloads?
A FIFO scheduler grants each job whatever single GPU is free next. What goes wrong for GPU workloads specifically?
FIFO is fair in ARRIVAL ORDER, but that's not the same as being workable for GPU jobs — a large distributed job needing 8 GPUs simultaneously can't make progress if it only ever gets offered ONE free GPU at a time between a stream of smaller jobs.
Two distinct problems: STARVATION (a big job never gets a chance to accumulate all the GPUs it needs at once, because small jobs keep consuming them one at a time) and FRAGMENTATION (even when enough total GPUs are free, they can be spread across the cluster in a way that's bad for a job that needs fast communication between its workers).
Scheduler compute overhead for a queue is a minor concern — the real failures are starvation of large jobs and fragmentation-driven poor placement, which are correctness/performance problems for the actual GPU workloads.
FIFO with "grab whichever GPU is free" creates two GPU-specific failures: starvation (a large multi-GPU job never gets to accumulate everything it needs while smaller jobs keep taking individual free GPUs) and fragmentation (available capacity spread inconveniently across the cluster, hurting distributed jobs' communication performance even once scheduled).
Why this piece earns its place
FIFO is worth building properly rather than skipping past, because its shape is what every later policy gets bolted onto: a loop that takes a snapshot of the cluster, walks the queue in some order, and either places a job or records why it did not. That second half is the one people leave out, and it is the one you want at three in the morning. A scheduling decision is unreproducible by nature — the cluster state that produced it is gone seconds later — so the only way to answer someone who wants to know why their job is still sitting there is a per-pass record of what was considered and what was passed over, written at the time. Settle the loop's trigger early too. A periodic tick is simple and puts a floor under how long a freed GPU sits unused; re-running on every release is prompt and re-scores the whole backlog under churn. Either way the snapshot has to be consistent, or two passes hand the same GPU to two jobs. And treat the submit schema as an API from day one: every team's launch script and CI pipeline hardcodes whatever the first version accepted, so a field you never collected is a field no later policy can read.
What the new pieces do
- Scheduler APIgateway
- Accepts job requests (how many GPUs, for how long, what priority) and orchestrates everything that decides where and when they actually run.
- GPU Inventorydata
- The real-time record of every GPU in the cluster and whether it's free, allocated, or reserved — what every scheduling decision ultimately checks against.
Step 2 · Gang scheduling
All the GPUs at once, or none of them
A distributed training job needs, say, 8 GPUs to even start — with only 5 allocated, the other 3 worker processes have nothing to talk to, and the job can't make progress at all. Should the scheduler allocate GPUs to it incrementally as they free up?
A distributed job needs 8 GPUs simultaneously to function at all. Allocate them one at a time as they free up?
For a distributed training job, partial allocation isn't partial progress — a job with only 5 of 8 required GPUs typically can't run AT ALL, since the missing workers break the distributed computation. Incremental allocation just holds resources hostage without enabling any actual work.
Gang scheduling treats a multi-GPU job's full resource requirement as one atomic unit: either all of it is allocated together, letting the job actually start, or none of it is, and it waits. This avoids the failure mode of holding some resources uselessly while waiting on the rest.
Distributed training jobs are typically not decomposable into independent single-GPU units without changing the actual training algorithm — the workers need to communicate with each other as part of one coordinated computation, not run as unrelated separate jobs.
Implement gang scheduling: a job's full resource requirement is treated as one atomic unit. It waits in the Pending Jobs queue until every GPU it needs is simultaneously available, then all of them are allocated together — never partially, which would hold resources without enabling any actual progress.
Why this piece earns its place
All-or-nothing has to cover the end of a job as well as the start. Eight workers allocated together are one thing while they run: if a single worker segfaults, the other seven keep their GPUs and keep spending their team's share while making no progress, because the computation they belonged to no longer exists. So a gang needs one lifecycle identity — a record the scheduler can fail as a unit and reclaim all eight GPUs from — rather than eight independent allocations that happen to share a name. Nothing downstream catches this for you, either: the survivors are live processes on live hardware, so anything looking for something dead finds them perfectly healthy, and only the gang's own membership record knows the job it belonged to is over. Gang scheduling also forces an admission check. A job asking for more GPUs than the cluster physically holds, or more than its team could ever be allocated, waits forever and is indistinguishable from ordinary starvation in the queue. Reject it at submit, with that reason attached, instead of accepting a request that can never be satisfied.
What the new pieces do
- Pending Jobsevents
- Jobs waiting for their FULL resource requirement to become available simultaneously — a multi-GPU job doesn't start with only some of its GPUs.
Step 3 · Topology-aware placement
Which GPUs, not just how many
8 GPUs are free cluster-wide — but they're scattered across 8 different physical nodes, each pair connected only by a relatively slow network link rather than a fast direct interconnect. Does it matter which 8 the job actually gets?
8 free GPUs exist, but scattered far apart across the cluster's topology. Does WHICH specific GPUs a job gets matter?
For single-GPU jobs this is true, but distributed training workers communicate constantly (gradient synchronization, activations) — GPUs connected by a fast direct interconnect (like NVLink) communicate far faster than GPUs that have to route through a slower network link between separate nodes.
A distributed job's effective performance depends heavily on how fast its workers can exchange data — placing all 8 of its GPUs on well-connected, nearby hardware can mean substantially faster training than the same 8 GPU-count scattered across poorly-connected nodes, even though "8 GPUs" is identical in both cases on paper.
The opposite is closer to true — placement matters LEAST for single-GPU jobs (which have no cross-GPU communication at all) and matters MOST for multi-GPU distributed jobs, where inter-worker communication is a core part of the workload.
Add a Placement Engine that considers cluster topology, not just raw GPU count: for a multi-GPU job, prefer allocating GPUs that are close together (same node, fast interconnect) over scattering them across poorly-connected nodes. The same "8 GPUs" can perform very differently in practice depending on how well-connected they are to each other.
Why this piece earns its place
The engine's hardest calls are about the jobs that do not care where they land. Drop one small job into the middle of an otherwise empty machine and it performs exactly as it would have anywhere, while destroying the only place an eight-GPU gang could have gone — nothing failed, utilization went up, and the cluster quietly lost the ability to run its largest job. So the policy for small jobs is really a packing policy: put them on machines that are already dirty and keep whole machines clean, even when a cleaner one is nearer to hand. Get that backwards and you manufacture, one GPU at a time, exactly the scattering Step 1 warned about, and gang scheduling spends its life waiting on a set that keeps being broken. The topology model is also data, and data drifts. Nodes get added, a link degrades, a host comes back from repair wired differently, and the engine goes on scoring confidently against the graph it was handed. Treat it like inventory — discovered from the nodes, refreshed, versioned — because an engine reasoning about a topology that no longer exists is worse than any-free-GPU, since you believe it.
What the new pieces do
- Placement Engineservice
- Decides WHICH specific GPUs a job gets, not just how many — accounting for interconnect locality so multi-GPU communication stays fast.
Step 4 · Priority and preemption
An urgent interactive request beats a long batch job
The cluster is fully allocated to long-running batch training jobs. A latency-sensitive interactive inference request arrives and needs a GPU right now. Should it simply wait in line behind hours of batch work?
The cluster is full of long batch jobs. An urgent, latency-sensitive request arrives. Should it wait in the normal queue?
Treating a latency-sensitive interactive request identically to an hours-long batch job ignores that the two have fundamentally different urgency — a strict FIFO queue would make interactive workloads unusably slow whenever the cluster is busy with batch work, which is most of the time on a well-utilized cluster.
Priority-based preemption reflects that not all work is equally urgent: an interactive request that a person or system is actively waiting on can reclaim GPUs from a batch job that has no similar time pressure, as long as the preempted job can cleanly resume later rather than losing its progress.
A statically reserved interactive-only pool sits idle whenever interactive demand is low, wasting expensive capacity that batch jobs could otherwise be using — dynamic preemption uses the SAME shared pool efficiently rather than partitioning it permanently.
Introduce a Preemptor: jobs carry a priority level, and a high-priority request can reclaim GPUs from a running lower-priority job, which is signaled to checkpoint its state and pause rather than being abruptly killed. This lets latency-sensitive work get served immediately from a SHARED pool, without permanently partitioning capacity.
Why this piece earns its place
Priority is a number a human types, so the real design work is deciding who may type what. Left open it only ratchets upward, because the one piece of evidence anybody has that their level was set right is that their jobs start — and once everything is urgent you are back to FIFO with extra steps, with no cheap way out, since demoting a team is a conversation rather than a config change. The ladder therefore belongs in config and needs an owner: what each level means, which groups may submit at it, whether it is permitted to preempt at all. Only the level itself rides on the request. The other thing the word "preempt" hides is that it rarely means one victim. Freeing eight well-placed GPUs usually means stopping several jobs on the same machine, and if you stop them one after another the GPUs freed first get taken by something else before the set is whole. The preemption and the allocation it exists to enable have to commit together, or you pay the full cost of interrupting work and still fail to start the job.
What the new pieces do
- Preemptorservice
- Reclaims GPUs from lower-priority jobs (checkpointing them first) to serve higher-priority, latency-sensitive requests immediately.
Step 5 · Fair-share across teams
Priority alone can let one team dominate
Multiple teams share the same GPU pool. If priority is the only allocation mechanism, could one team flood the scheduler with high-priority jobs and effectively starve every other team, even without doing anything technically against the rules?
Multiple teams compete for the same shared GPU pool, with only priority deciding allocation. What's the risk?
Priority determines WHICH job runs next in a given moment, but says nothing about aggregate fairness across teams over time — a team that simply submits more high-priority jobs than others could legitimately, within the rules, consume a wildly disproportionate share of total cluster capacity.
Priority alone optimizes for "serve the most urgent thing right now" but has no concept of "and make sure this doesn't let one team consume everyone else's share over time" — that requires a SEPARATE mechanism, fair-share quotas, layered on top of priority.
The starvation risk exists even under entirely good-faith, policy-compliant usage — a team simply having more urgent work than others, with no malicious intent, can still exhaust shared capacity under a priority-only allocation scheme.
Layer fair-share quotas on top of priority: each team gets a guaranteed baseline share of cluster capacity (GPU-hours, or a percentage), and even legitimate high-priority demand from one team is capped relative to that quota, protecting other teams' access. Priority decides ordering WITHIN what a team is entitled to; fair-share decides how much each team is entitled to in the first place.
Why this piece earns its place
The interesting choice is what the quota is denominated in, and the options behave nothing alike. A cap on concurrent GPUs is simple to explain and lets a team sit at its ceiling indefinitely. A budget of GPU-hours over a fixed window lets them burst far above that ceiling and then go quiet, and turns every window boundary into a stampede as everyone's balance resets at the same instant. Usage with a decay — older consumption counting for less as it ages — avoids both the cliff and the permanent penalty. Then decide what a job is charged to and enforce it at submit rather than trusting a field the submitter fills in, or the policy is advisory and the numbers in the report are fiction. The consequence nobody plans for is that a team is not the last level. Inside one team's share, whoever submits first takes all of it, and the crowding-out this step exists to stop reappears between colleagues with nothing left to catch it. So the quota wants to be a tree with weights rather than one flat number per team — and the moment it is a tree, every node in it needs somebody who owns the split.
- per-team quotabaseline guaranteed share
- prioritydecides order within that share
- fair-shareprotects against legitimate flooding
What the new pieces do
- Fair-Share Quotasdata
- Caps how much of the shared cluster any single team can consume, even under legitimate high-priority demand — preventing one team from starving everyone else.
Step 6 · Checkpoint, don't restart
A preempted job shouldn't lose hours of work
A batch training job running for 6 hours gets preempted by a high-priority interactive request. If it simply loses all progress and has to restart from scratch when it resumes, preemption becomes extremely costly for whoever's work keeps getting bumped.
Preemption signals the job to checkpoint its current state (model weights, optimizer state, training step) to durable storage before its GPUs are reclaimed — not an abrupt kill. When capacity frees up again, the job resumes FROM that checkpoint, continuing roughly where it left off rather than restarting from step zero. This makes preemption's real cost the checkpoint/resume overhead (typically minutes), not the hours of lost progress a hard kill would cause.
Why this piece earns its place
The guarantee is only as strong as the deadline attached to it. Preemption is a signal plus a grace window, and a job that cannot finish writing inside that window is killed where it stands and comes back from its last periodic save instead — so the window has to be a published number rather than an internal one, because it is what framework authors size their save interval against. Leave it undocumented and every preemption becomes a lottery over how much recent work survives. The failure that teaches this is a checkpoint storm: a preemption wave sends many jobs writing their state to the same durable store in the same minute, so write time climbs precisely when the scheduler is leaning on preemption hardest, and jobs that normally save comfortably start missing the window. Preemption cost is therefore coupled to storage contention, which nothing in the scheduler models. Worth knowing too: sharded state is usually written per worker, so a job checkpointed across eight workers resumes onto eight unless the framework can reshard — the checkpoint constrains the shape of the resume, not only the count.
What the new pieces do
- Checkpointsdata
- Saved job state that lets a preempted job resume from where it left off instead of starting its (often hours-long) work over from scratch.
Step 7 · Elastic scaling and backfill
Use idle GPUs opportunistically
Some training jobs can dynamically grow or shrink their worker count (elastic training). If GPUs are briefly idle waiting for the next gang-scheduled job to accumulate its full requirement, is there useful work they could be doing in the meantime?
For jobs that support elastic scaling, the scheduler can opportunistically grow them onto temporarily-idle GPUs (backfilling capacity that would otherwise sit unused while waiting for the next gang-scheduled allocation) and shrink them back down when that capacity is needed elsewhere. This squeezes real utilization out of gaps the gang-scheduling and preemption policies would otherwise leave idle.
Why this piece earns its place
Resizing a job changes the run, not just the schedule. World size sets the effective batch size, which the learning-rate schedule and the step count per epoch were tuned against, so a job quietly grown mid-training produces a result its owner cannot reproduce and may not be able to explain. That makes elasticity opt-in, with a declared minimum and maximum the framework has agreed to handle, rather than something the scheduler does on its own because capacity appeared. Shrink is the half the scheduler should care about: borrowed GPUs are only useful if they come back on the timescale a waiting gang needs them, so a resize that costs as much as a checkpoint is not reclaimable capacity, it is a second squatter. And the feature lives or dies on accounting. If opportunistic hours are billed against a team's guaranteed share at the same rate as scheduled work, no team enables elasticity, and the utilization this whole step exists to recover never materializes.
Step 8 · The sharp edges
Zombie processes and scheduler drift
A job crashes ungracefully without releasing its GPU memory — the scheduler's bookkeeping says those GPUs are still allocated, but nothing useful is actually running on them. Over time, does the scheduler's view of the cluster stay accurate?
Run continuous health checks that verify actual GPU utilization and process health against the scheduler's own allocation records, forcibly reclaiming GPUs held by crashed or zombie processes that never properly released them. This closes the gap between what the scheduler THINKS is allocated and what's actually true on the cluster — a reconciliation loop, not a one-time assumption that bookkeeping stays accurate forever.
Why this piece earns its place
The word doing the work here is reclaim, and it means more than editing a row. Mark the GPUs free while a dead job's memory is still resident, and the next job to land there fails on allocation with an error that looks entirely like its own fault — the same drift, now wearing somebody else's incident. So reclamation is two operations rather than one: clear away what is left of the old process, then confirm the device memory genuinely came back before those GPUs return to inventory. Where that confirmation cannot be obtained, the honest state is quarantined, not free. That also settles the case where a node has simply stopped answering, which from the scheduler's side is indistinguishable from a node that died. Treat silence as proof and you will eventually place a second job on top of one that is running fine behind a network fault, and two jobs sharing memory each assumed was theirs lose more work than the zombie ever cost. Close the allocation after a bounded wait so its owner stops being billed for it, but leave the hardware out of service until the node checks back in clean. And put every reclamation in front of the job's owner, because from their side the job simply vanished.
You did it
You just designed a GPU cluster scheduler.
Everything you assembled, in order
- FIFO with "grab any free GPU" starves large distributed jobs and fragments capacity — GPU scheduling needs coordinated, atomic allocation.
- Gang scheduling allocates a multi-GPU job's full requirement atomically, all at once, since partial allocation often enables no progress at all.
- Topology-aware placement matters because WHICH specific GPUs a distributed job gets — not just how many — directly affects real training throughput.
- Priority-based preemption lets urgent, latency-sensitive work reclaim GPUs from lower-priority jobs immediately, from a shared pool.
- Fair-share quotas layer on top of priority to prevent one team's legitimate high-priority volume from starving every other team over time.
- Checkpointing makes preemption cheap for the preempted job — resume from where it left off, not restart from scratch.
- Elastic jobs can opportunistically backfill idle capacity that gang scheduling and preemption would otherwise leave temporarily unused.
- Continuous health checks reconcile the scheduler's bookkeeping against real cluster state, reclaiming GPUs from crashed or zombie processes.
