The big idea

GPUs are not fungible, interchangeable slots

A generic job scheduler assumes any worker can run any job. GPU scheduling breaks that assumption in several specific ways: distributed training needs MANY GPUs allocated at once, not incrementally; which specific GPUs matter (interconnect distance affects real performance); and GPUs are so expensive that fair allocation across competing teams is a first-class requirement, not an afterthought.

We'll build the scheduler around these specific constraints: all-or-nothing gang scheduling, topology-aware placement, priority-based preemption with checkpointing, and fair-share quotas so scarce capacity doesn't silently concentrate in one team's hands.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill the placement engine and flood a burst of interactive jobs to see what degrades and what stays fair. Hit Begin.

Step 1 · The baseline

FIFO, first free GPU

Simplest scheduler: jobs are served in the order they arrive, each grabbing whatever GPU is currently free. What breaks specifically for GPU workloads?

Design decision: A FIFO scheduler grants each job whatever single GPU is free next. What goes wrong for GPU workloads specifically?

The call: A job needing many GPUs at once can be starved indefinitely if smaller jobs keep claiming individual free GPUs before enough accumulate for the large job — and jobs can end up fragmented across GPUs with poor interconnect locality, hurting distributed-training performance even when they do get scheduled. — Two distinct problems: STARVATION (a big job never gets a chance to accumulate all the GPUs it needs at once, because small jobs keep consuming them one at a time) and FRAGMENTATION (even when enough total GPUs are free, they can be spread across the cluster in a way that's bad for a job that needs fast communication between its workers).

FIFO with "grab whichever GPU is free" creates two GPU-specific failures: starvation (a large multi-GPU job never gets to accumulate everything it needs while smaller jobs keep taking individual free GPUs) and fragmentation (available capacity spread inconveniently across the cluster, hurting distributed jobs' communication performance even once scheduled).

GPU scheduling has different constraints than generic job scheduling: A generic job scheduler treats "one free worker" as sufficient to make progress. GPU workloads, especially distributed training, need a coordinated set of resources allocated together and placed thoughtfully — a fundamentally different scheduling problem.

Step 2 · Gang scheduling

All the GPUs at once, or none of them

A distributed training job needs, say, 8 GPUs to even start — with only 5 allocated, the other 3 worker processes have nothing to talk to, and the job can't make progress at all. Should the scheduler allocate GPUs to it incrementally as they free up?

Design decision: A distributed job needs 8 GPUs simultaneously to function at all. Allocate them one at a time as they free up?

The call: Use gang scheduling: hold the job in a Pending queue until ALL the GPUs it requires are simultaneously available, then allocate them atomically together — never partially. — Gang scheduling treats a multi-GPU job's full resource requirement as one atomic unit: either all of it is allocated together, letting the job actually start, or none of it is, and it waits. This avoids the failure mode of holding some resources uselessly while waiting on the rest.

Implement gang scheduling: a job's full resource requirement is treated as one atomic unit. It waits in the Pending Jobs queue until every GPU it needs is simultaneously available, then all of them are allocated together — never partially, which would hold resources without enabling any actual progress.

Atomic allocation for coordinated work: Whenever a job's components MUST all be present together to do anything useful (distributed training workers that need to communicate), the scheduler has to allocate atomically — this is the same all-or-nothing principle as an atomic database transaction, applied to cluster resource allocation.

Step 3 · Topology-aware placement

Which GPUs, not just how many

8 GPUs are free cluster-wide — but they're scattered across 8 different physical nodes, each pair connected only by a relatively slow network link rather than a fast direct interconnect. Does it matter which 8 the job actually gets?

Design decision: 8 free GPUs exist, but scattered far apart across the cluster's topology. Does WHICH specific GPUs a job gets matter?

The call: Yes — the Placement Engine should prefer allocating GPUs that are topologically close together (same node, fast interconnect) for a single distributed job, since communication speed between the job's workers directly affects real training throughput. — A distributed job's effective performance depends heavily on how fast its workers can exchange data — placing all 8 of its GPUs on well-connected, nearby hardware can mean substantially faster training than the same 8 GPU-count scattered across poorly-connected nodes, even though "8 GPUs" is identical in both cases on paper.

Add a Placement Engine that considers cluster topology, not just raw GPU count: for a multi-GPU job, prefer allocating GPUs that are close together (same node, fast interconnect) over scattering them across poorly-connected nodes. The same "8 GPUs" can perform very differently in practice depending on how well-connected they are to each other.

Count isn't the whole resource specification: For workloads with significant inter-worker communication, WHERE a resource lives relative to its peers is part of the real specification, not an implementation detail — treating "8 GPUs" as a fungible count while ignoring their relative placement misses a major real-world performance factor.

Step 4 · Priority and preemption

An urgent interactive request beats a long batch job

The cluster is fully allocated to long-running batch training jobs. A latency-sensitive interactive inference request arrives and needs a GPU right now. Should it simply wait in line behind hours of batch work?

Design decision: The cluster is full of long batch jobs. An urgent, latency-sensitive request arrives. Should it wait in the normal queue?

The call: Give jobs a priority level, and let a high-priority request PREEMPT (checkpoint and pause) a lower-priority running job to free up GPUs immediately — the preempted job resumes later from its checkpoint. — Priority-based preemption reflects that not all work is equally urgent: an interactive request that a person or system is actively waiting on can reclaim GPUs from a batch job that has no similar time pressure, as long as the preempted job can cleanly resume later rather than losing its progress.

Introduce a Preemptor: jobs carry a priority level, and a high-priority request can reclaim GPUs from a running lower-priority job, which is signaled to checkpoint its state and pause rather than being abruptly killed. This lets latency-sensitive work get served immediately from a SHARED pool, without permanently partitioning capacity.

Not all work has the same urgency: Treating every job identically regardless of how time-sensitive it actually is wastes the scheduler's ability to make the cluster feel responsive for what needs to be — preemption is how a shared resource pool serves urgent and non-urgent work well, without static, wasteful partitioning.

Step 5 · Fair-share across teams

Priority alone can let one team dominate

Multiple teams share the same GPU pool. If priority is the only allocation mechanism, could one team flood the scheduler with high-priority jobs and effectively starve every other team, even without doing anything technically against the rules?

Design decision: Multiple teams compete for the same shared GPU pool, with only priority deciding allocation. What's the risk?

The call: A team that submits a large enough volume of high-priority jobs can legitimately, within the rules, dominate the shared pool — leaving other teams with far less than their fair share, even without anyone doing anything against policy. — Priority alone optimizes for "serve the most urgent thing right now" but has no concept of "and make sure this doesn't let one team consume everyone else's share over time" — that requires a SEPARATE mechanism, fair-share quotas, layered on top of priority.

Layer fair-share quotas on top of priority: each team gets a guaranteed baseline share of cluster capacity (GPU-hours, or a percentage), and even legitimate high-priority demand from one team is capped relative to that quota, protecting other teams' access. Priority decides ordering WITHIN what a team is entitled to; fair-share decides how much each team is entitled to in the first place.

Priority and fairness are two different axes: "What runs next" (priority) and "how much does each stakeholder get over time" (fair-share) are genuinely separate concerns — optimizing only for the first, without the second, systematically advantages whoever submits the most or highest-priority work, regardless of whether that's actually the intended allocation policy.

Step 6 · Checkpoint, don't restart

A preempted job shouldn't lose hours of work

A batch training job running for 6 hours gets preempted by a high-priority interactive request. If it simply loses all progress and has to restart from scratch when it resumes, preemption becomes extremely costly for whoever's work keeps getting bumped.

Preemption signals the job to checkpoint its current state (model weights, optimizer state, training step) to durable storage before its GPUs are reclaimed — not an abrupt kill. When capacity frees up again, the job resumes FROM that checkpoint, continuing roughly where it left off rather than restarting from step zero. This makes preemption's real cost the checkpoint/resume overhead (typically minutes), not the hours of lost progress a hard kill would cause.

Make the expensive operation cheap to interrupt: Preemption is only a viable scheduling tool if being preempted is CHEAP for the preempted job — checkpointing is the mechanism that makes "pause and resume later" close to free relative to "lose everything and start over," which is what actually makes Step 4's preemption policy acceptable to the teams whose jobs get preempted.

Step 7 · Elastic scaling and backfill

Use idle GPUs opportunistically

Some training jobs can dynamically grow or shrink their worker count (elastic training). If GPUs are briefly idle waiting for the next gang-scheduled job to accumulate its full requirement, is there useful work they could be doing in the meantime?

For jobs that support elastic scaling, the scheduler can opportunistically grow them onto temporarily-idle GPUs (backfilling capacity that would otherwise sit unused while waiting for the next gang-scheduled allocation) and shrink them back down when that capacity is needed elsewhere. This squeezes real utilization out of gaps the gang-scheduling and preemption policies would otherwise leave idle.

Opportunistic use of transient capacity: Not every job can be elastic, but for the ones that can, treating brief idle windows as usable capacity (rather than simply unused) meaningfully improves overall cluster utilization — the same principle as backfill scheduling in HPC clusters, applied to GPU training jobs specifically.

Step 8 · The sharp edges

Zombie processes and scheduler drift

A job crashes ungracefully without releasing its GPU memory — the scheduler's bookkeeping says those GPUs are still allocated, but nothing useful is actually running on them. Over time, does the scheduler's view of the cluster stay accurate?

Run continuous health checks that verify actual GPU utilization and process health against the scheduler's own allocation records, forcibly reclaiming GPUs held by crashed or zombie processes that never properly released them. This closes the gap between what the scheduler THINKS is allocated and what's actually true on the cluster — a reconciliation loop, not a one-time assumption that bookkeeping stays accurate forever.

Design for the unhappy path: Placement engine down → degrades to less-optimal placement, doesn't stop scheduling. Interactive flood → preemption plus fair-share keeps it bounded and fair. Crashed jobs → active reconciliation catches drift between bookkeeping and reality. A scheduler that only works when every job exits cleanly is a demo; one that reconciles against real cluster state is a product.

You did it

You just designed a GPU cluster scheduler.

• F — I
• G — a
• T — o
• P — r
• F — a
• C — h
• E — l
• C — o