The big idea
Why is running code hard?
An AI agent writes code to solve a task and needs to run it. An online judge accepts a user's submission. A notebook executes a cell. In every case you're about to execute code you did not write and cannot trust — it could try to read secrets, attack internal systems, exfiltrate data, or hog the whole machine. And you have to do it fast, at scale, for many concurrent runs.
A code execution sandbox runs untrusted code safely: strong isolation (so code can't touch anything it shouldn't), strict resource limits (so it can't exhaust the machine), ephemeral environments (a fresh sandbox per run, destroyed after), and a warm pool so it's still fast. Security and speed are the whole problem — the "run code" part is easy; doing it without getting owned is not.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end drop the isolation and remove resource limits to see the two ways untrusted code hurts you. Hit Begin.
Step 1 · Never run it directly
Untrusted code = full compromise
The naive approach: just exec the code in your service process (or on the host). Why is this catastrophic when the code is untrusted?
Design decision: Run untrusted (LLM-generated) code directly in your service. What's the danger?
The call: It might have a syntax error. — Syntax errors are harmless. The real danger is that untrusted code running with your privileges can do anything your service can — read secrets, exfiltrate data, attack internal systems, or take down the host.
Executing untrusted code in your process means it runs with your privileges and environment — it can read secrets and env vars, touch the filesystem and other users' data, consume all CPU/memory, and call internal services. That's arbitrary code execution on your own infrastructure: a complete compromise. The code must run somewhere with no ambient access to anything that matters — isolated from the host, the network, and every other run.
Assume the code is hostile: The core mindset: treat every run as potentially malicious, because LLM-generated code can be wrong or manipulated and user submissions can be adversarial. You don't sanitize your way to safety — you contain execution so that even fully malicious code can't reach anything valuable.
Step 2 · A service around execution
Submit, run isolated, capture
Wrap execution behind a service. What's the basic contract, regardless of the isolation tech?
An execution API takes code + language + inputs, runs it inside an isolated sandbox under strict limits, and returns only the captured result — stdout/stderr, return value, exit status, and metrics. Nothing else crosses the boundary: the sandbox has no access in (no secrets, no network) and only structured output goes out. This clean submit→isolate→capture contract is the frame; the next steps make the isolation and limits real.
A one-way boundary: The sandbox is a sealed box: untrusted code goes in, only captured output comes out. Designing the contract this way — no ambient inputs, only structured results — means the blast radius of anything the code does is confined to a disposable environment.
Step 3 · Contain what it can touch
Isolation: containers & microVMs
The code must run somewhere it can't harm anything. What actually provides that isolation, and how strong does it need to be?
Design decision: What isolates untrusted code from the host and other runs?
The call: A hardened sandbox — containers with restricted syscalls (e.g. gVisor/seccomp) or lightweight microVMs (e.g. Firecracker) — with no network and least privilege. — Untrusted code runs inside a container hardened with syscall filtering/user-namespacing, or a lightweight microVM that gives near-VM isolation with fast startup. Deny network by default, drop privileges, restrict the filesystem, and layer defenses (defense in depth) so a single weakness doesn't breach the host.
Run the code inside a hardened sandbox: a container with restricted syscalls (seccomp/gVisor-style filtering, user namespaces) or a lightweight microVM (Firecracker-style) that gives near-VM isolation with fast boot. Apply least privilege — no network by default, a minimal read-only-ish filesystem, dropped capabilities — and layer these (defense in depth) so no single weakness breaches the host. Isolation is enforced by the runtime, never by trusting the code.
Containers vs microVMs: Containers share the host kernel, so a kernel bug can be an escape — hardened with seccomp/gVisor they're much safer. MicroVMs (Firecracker) run a tiny guest kernel for near-VM isolation with millisecond boots, a stronger boundary at slightly more overhead. High-risk multi-tenant execution leans toward microVMs; both use least privilege + no network.
Step 4 · Contain how much it consumes
Resource limits
Isolation stops code from reaching things — but code can still hang forever, allocate all memory, spawn endless processes (a fork bomb), or fill the disk, taking down the worker and co-located runs. How do you bound consumption?
Enforce strict resource limits per run: a CPU quota, a memory cap, a hard wall-clock timeout, a process/thread limit (to stop fork bombs), and disk quotas — via cgroups/ulimits or the microVM's allocation. Anything exceeding its limits is hard-killed. This turns "runs forever / eats the machine" into "fails cleanly at its quota", protecting the worker and every other tenant. Isolation contains what; limits contain how much — you need both.
Quotas + a hard kill: Every run gets a resource budget and a deadline; exceed either and it's terminated. cgroups cap CPU/memory/pids, ulimits cap files/processes, and a wall-clock timeout kills hangs. Without this, one infinite loop or fork bomb is a denial-of-service against the whole fleet.
Step 5 · The execution pipeline
Queue → sandbox → capture
Put it together into a flow that handles many concurrent runs and returns results. What's the path a submission takes?
Submissions flow through a queue to an orchestrator that assigns each to a fresh isolated sandbox (with limits wired up), runs the code, captures stdout/stderr/return/exit/metrics, returns the result, and tears the sandbox down. The queue absorbs bursts and lets you scale workers; the orchestrator manages sandbox lifecycle and enforces limits; only the structured result leaves. Long or interactive runs stream output; batch judge runs return when done.
Lifecycle: assign, run, capture, destroy: The orchestrator owns the sandbox lifecycle: pull a clean sandbox, execute under limits, collect only the allowed output, and reclaim/destroy it. Queueing decouples bursty demand from finite sandbox capacity, exactly like other heavy-job systems.
Step 6 · Nothing persists
Ephemeral sandboxes
If a sandbox were reused across untrusted runs without resetting, one run could leave behind malware, state, or traps for the next. How do you prevent cross-run contamination?
Design decision: How do you stop one untrusted run from affecting the next in the same sandbox?
The call: Reuse one long-lived sandbox and clean up files between runs. — Manually cleaning a reused environment is error-prone and leaves subtle state (installed packages, background processes, kernel state) that untrusted code can exploit against the next run. A fresh, destroyed-after sandbox is the safe default.
Make sandboxes ephemeral: a fresh environment per run (or per user session), destroyed afterward, never reused with dirty state. Nothing an untrusted run does — files, leftover processes, memory, installed packages — can persist to affect another run or user. For workflows that need continuity (an agent iterating, a notebook), keep a per-session sandbox that lives only for that session and is still isolated and disposable — persistence is scoped to one trust boundary, then wiped.
Disposable by default: Ephemerality guarantees isolation across time, not just at a single instant: a clean sandbox in, a destroyed sandbox out, so cross-run/cross-tenant contamination is impossible by construction. Stateful sessions are allowed but scoped to a single user and torn down at the end.
Step 7 · Fast enough to use
Warm pools & scaling
Booting a fresh microVM/container per run adds startup latency, and an agent that runs code in a loop can't wait seconds each time. How do you keep it fast while staying ephemeral?
Keep a warm pool of pre-booted, ready sandboxes so a run starts in milliseconds instead of waiting for a cold boot — hand one out, run, discard it, and replenish the pool in the background. Autoscale the pool and worker fleet on demand. MicroVMs like Firecracker are chosen partly because they boot in ~milliseconds, making fresh-per-run affordable. Balance reuse vs isolation carefully: pre-warming the environment is fine, but never reuse one that has already run untrusted code.
Warm, not reused: The trick is pre-booting clean sandboxes (warm) without reusing dirty ones. You amortize startup by keeping a ready supply, so ephemeral execution stays low-latency. Fast-booting microVMs make "new sandbox every run" practical at interactive speeds.
Step 8 · The sharp edges
Escapes, egress & real workloads
Even with isolation and limits, the hard parts remain: sandbox escapes, controlling network egress (data exfiltration), letting real code install packages or access data it legitimately needs, and abuse.
Assume escapes are possible and defend in depth — keep hosts patched, minimize the guest, run untrusted workloads on isolated infrastructure, and monitor. Control network egress tightly: default-deny, and if code needs the internet, allowlist/proxy it — uncontrolled egress means data exfiltration. Handle package installs / data access via a controlled path (a vetted mirror, mounted read-only inputs) rather than open network. Add rate limits, quotas and monitoring to stop abuse (crypto-mining, DoS). And log/observe runs (without weakening isolation) so you can detect and respond.
Design for the unhappy path: Escape → defense in depth + patched, minimal, isolated hosts. Exfiltration → default-deny egress, allowlist/proxy. Real needs → controlled package/data paths, not open net. Abuse → quotas + monitoring. Running untrusted code safely is never "done" — it's layered containment plus vigilance, because the threat is adversarial by nature.
You did it
You just designed a code execution sandbox.
- Running untrusted (LLM-generated or user) code directly is arbitrary code execution on your infra — a full compromise.
- Wrap execution in a service: code in, isolated run, only captured result out — a one-way boundary.
- Isolate with hardened containers (restricted syscalls) or microVMs, least privilege, no network by default.
- Enforce resource limits (CPU/memory/time/processes/disk) and hard-kill overruns to stop hangs and fork bombs.
- Pipeline: queue → orchestrator assigns a fresh sandbox under limits → capture output → destroy.
- Make sandboxes ephemeral (fresh per run/session, destroyed) so nothing contaminates the next run.
- Keep a warm pool for millisecond starts; control egress, package/data access, and abuse — defense in depth.