The big idea
What is an online judge, specifically?
Users submit arbitrary code — sometimes buggy, sometimes deliberately malicious — and expect it to be run, measured, and graded, at contest scale, fairly, without ever letting one submission compromise the host, starve other users, or get an inconsistent verdict on a rerun.
We'll build it around the one property that has to hold above all others: untrusted code execution must never become a security or fairness incident. Everything — sandboxing, queueing, resource limits, fair scheduling — flows from that.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill the sandbox workers and flood a contest-start burst to see what survives. Hit Begin.
Step 1 · The baseline
Run the code right here
Simplest version: the API server receives a submission and just... runs it, directly, in-process or as a plain subprocess, to get the output. Why is this dangerous?
Design decision: The API server directly executes arbitrary user-submitted code. What's the danger?
The call: It's a full arbitrary-code-execution vulnerability on shared, production infrastructure: a malicious submission could read other users' data, attack the network, exhaust host resources, or compromise the server outright. — Running untrusted code with no isolation on the same infrastructure serving every other user's requests means a single malicious submission can potentially read filesystem data it shouldn't, make outbound network calls, fork-bomb the host, or otherwise compromise a shared production system — this isn't a hypothetical, it's the defining risk of the entire product category.
Running submitted code directly, with no isolation, on infrastructure shared with everything else is a full arbitrary-code-execution vulnerability. Nothing else in this design matters if this foundational problem isn't solved first: untrusted code must never run unsandboxed on shared infrastructure.
This system's entire job is running untrusted code safely: Unlike most systems where security is one concern among many, an online judge's core PRODUCT FUNCTION is executing arbitrary, potentially adversarial code — security isn't a bolt-on feature here, it's the central engineering problem.
Step 2 · Real sandboxing
A container alone isn't enough
Isolate execution in some kind of sandbox — a container, a VM, a purpose-built sandbox runtime. But a plain container with default settings still shares the host kernel, has no CPU/memory ceiling by default, and can still open network connections or fork endlessly. What does a sandbox actually need to guarantee?
Design decision: What does "sandboxed execution" actually need to guarantee, beyond just running in a container?
The call: Hard CPU-time and memory limits (via cgroups), no network access, filesystem access limited to only what's needed, and process/fork limits — combined, not any single one alone — plus destroying the sandbox environment after every single run. — Real isolation needs multiple, layered guarantees together: bounded CPU time (kills runaway loops), bounded memory (kills memory exhaustion), no network (prevents exfiltration or attacking other systems), restricted filesystem access, and a cap on process/thread creation (prevents fork bombs) — and the whole environment is thrown away after each run so nothing persists between submissions.
Real sandboxing means several guarantees enforced together: hard CPU-time and memory limits via cgroups, no network access, minimal filesystem access, a cap on process/fork creation (stopping fork bombs), and a fully ephemeral environment destroyed after every single run so nothing persists or leaks between submissions.
Defense in depth: No single isolation mechanism is sufficient on its own against a sufficiently adversarial submission — the guarantees have to be layered, so a gap in any one (say, a container-escape bug) is still contained by the others (no network, capped resources, ephemeral teardown).
Step 3 · Async execution at contest scale
Submission and execution are two different moments
Execution can take real time — seconds, sometimes more for slow submissions — and during a live contest, thousands of submissions can land within the same few seconds. Should the API request block until execution completes?
Design decision: A contest produces thousands of near-simultaneous submissions, each taking real time to execute. Block the request until done?
The call: Accept the submission immediately, place it on a durable Job Queue, and let a pool of Sandbox Workers pull from that queue asynchronously — the client polls or gets pushed the verdict once judging completes. — Decoupling "accepted" from "judged" via a durable queue means the API can accept submissions as fast as they arrive, completely independent of how many workers are currently busy executing code. Worker capacity can scale up during a contest burst and down afterward, without the API path itself ever being the bottleneck.
Accept submissions immediately and place them on a durable Job Queue; a pool of Sandbox Workers pulls and executes asynchronously, independent of submission rate. This decouples "accepted" from "judged" — the API stays fast and available even during a massive contest-start burst, and worker capacity scales to match demand.
Decouple the fast path from the slow path: The same pattern as ride matching's batch window and the logging pipeline's local buffering: whatever takes real time (sandboxed execution, here) shouldn't block the fast, cheap operation (accepting a submission) — a queue in between lets each scale independently.
Step 4 · Determining the verdict
Hidden tests, checkers, and short-circuiting
A submission runs against multiple hidden test cases. Some problems have exactly one correct output per input (exact match); others have multiple valid outputs (floating-point tolerance, multiple valid orderings). How does the judge decide pass or fail?
Run the submission against each hidden test case in Test Cases, comparing actual output to expected — either an exact match, or, for problems where multiple outputs can be valid, a custom checker program that validates correctness rather than doing a literal diff. The judge typically short-circuits on the first failing test (reporting that specific verdict — Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error) rather than always running every test, saving compute on submissions that are already known to fail.
Not every correctness check is a string comparison: For many real problems, "correct" isn't a single fixed string — a checker program encodes the actual correctness CRITERION (does this satisfy the constraints, is it within floating-point tolerance) rather than relying on exact text matching, which would incorrectly reject valid alternative answers.
Step 5 · Precise, fair resource limits
CPU time, not wall-clock
Enforce a time limit on each submission. If the judge is under heavy load (many workers executing simultaneously, contending for the same host's CPU), a wall-clock timeout can unfairly penalize a submission that's algorithmically fine but simply got less CPU time due to contention. What should the time limit actually measure?
Design decision: The judge is under heavy contest load. What should the time limit for each submission actually measure?
The call: CPU time actually consumed by the process (via cgroups CPU accounting), independent of how much wall-clock time passed while it may have been waiting for a CPU slice. — CPU-time limits (enforced via cgroups) measure the actual computational work done, regardless of how contended the host was at that moment — a submission that needs 500ms of CPU work gets judged on that 500ms whether it ran alone or alongside a thousand other contest submissions, keeping verdicts fair and consistent.
Enforce limits on actual CPU time consumed (via cgroups CPU accounting), not wall-clock duration. This keeps a submission's verdict consistent and fair regardless of how contended the host happens to be at that moment — the same algorithmic solution gets the same time-limit outcome whether it runs alone or during the busiest minute of a contest.
Measure what you actually care about: "How much real computational work did this do" (CPU time) is the meaningful, fair signal for judging efficiency; "how long did a human waiting on a clock perceive it took" (wall-clock time) is contaminated by factors — host load, scheduling — that have nothing to do with the submission's own correctness or efficiency.
Step 6 · Determinism
The same submission should get the same verdict, always
A judge that gives a different verdict on a rerun of the exact same code — sometimes Accepted, sometimes a flaky Time Limit Exceeded — destroys trust, especially in a competitive contest setting where a verdict has real stakes.
CPU-time limiting (Step 5) already removes the dominant source of flakiness — host contention no longer affects the measured time. Combine it with fully ephemeral, freshly-provisioned sandboxes per run (no shared, potentially-degraded state carried between submissions) and, for genuinely borderline cases, an explicit rejudge mechanism that reruns a submission under controlled conditions if its verdict is disputed. Determinism isn't achieved by hoping the system behaves consistently — it's achieved by removing every source of non-determinism one at a time.
Determinism is engineered, not assumed: A judge's credibility depends on "the same input always produces the same output." Every design decision so far (CPU-time not wall-clock, ephemeral sandboxes, isolated resource limits) is, among other things, in service of this one property.
Step 7 · Fair queueing under contest load
One user shouldn't starve everyone else
A contest opens. Some users submit rapidly (retrying slightly different solutions), others submit once and wait. Should the Job Queue be strictly first-come-first-served across ALL submissions from ALL users?
Design decision: During a contest, should worker capacity be allocated strictly first-come-first-served across everyone's submissions?
The call: Give each user a fair, bounded share of worker throughput — a per-user queue or rate limit within the overall queue — so one user's submission volume can't crowd out everyone else's judging turnaround during a shared contest window. — Per-user fairness (a fair-queueing discipline, or a rate limit on how many of one user's submissions can be "in flight" being judged at once) ensures worker capacity is distributed roughly evenly across active users, regardless of how many times any single user has submitted — closer to what contest participants actually expect as "fair."
Implement fair queueing: each user gets a bounded share of worker throughput — a per-user submission-in-flight cap or a fair-share scheduling discipline within the overall queue — so rapid resubmission by one user can't crowd out judging turnaround for everyone else during a shared, high-stakes contest window. Combine this with worker autoscaling to absorb the aggregate burst.
Global throughput and per-user fairness are different goals: Maximizing total submissions-judged-per-second (global throughput) and ensuring no single user dominates that throughput (per-user fairness) can conflict — a fair-queueing discipline is specifically designed to balance both, rather than optimizing raw throughput alone at fairness's expense.
Step 8 · The sharp edges
Escape attempts, queue backlogs, and cleanup
Real-world stress tests: a submission deliberately trying to escape the sandbox, a queue backlog large enough that even fair-queued wait times become noticeable, and making sure every sandbox is genuinely, completely cleaned up between runs.
Defense in depth (Step 2) means an escape attempt has to defeat CPU/memory limits, no-network, restricted filesystem access, AND process caps simultaneously — a much higher bar than defeating any single mechanism. Queue backlogs are absorbed by worker autoscaling first, and, if a backlog still grows past a threshold, users see an honest estimated wait rather than a silent stall. And every sandbox is provisioned fresh and fully destroyed — not reset, not reused — after each run, so there's no path for one submission's leftover state to affect the next.
Design for the unhappy path: Worker outage → queue holds everything safely, nothing lost. Contest burst → fair queueing plus autoscaling. Escape attempt → layered isolation, not a single point of failure. A judge that only works for well-behaved code under light load is a demo; one that stays secure, fair, and consistent under adversarial and contest-scale conditions is a product.
You did it
You just designed an online judge.
- Running untrusted code unsandboxed on shared infrastructure is a full security vulnerability — nothing else matters until this is solved.
- Real sandboxing layers CPU/memory limits, no network access, restricted filesystem access, and process caps together, with a fully ephemeral environment per run.
- A durable job queue decouples "submission accepted" from "execution finished," letting worker capacity scale independently of submission rate.
- Verdicts come from comparing output against hidden test cases — either exact match or a checker program for problems with multiple valid outputs.
- CPU-time limits (via cgroups), not wall-clock time, keep verdicts fair and consistent regardless of host contention.
- Determinism is engineered by removing sources of non-determinism one at a time — CPU-time limiting, ephemeral sandboxes, and rejudge for disputed cases.
- Fair, per-user queueing prevents one user's submission volume from starving everyone else's judging turnaround during a live contest.
