The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
The environment is the training signal
The reasoning models everyone celebrates are trained by RL — but the algorithm gets too much credit. What actually teaches the model is the environment: which tasks it attempts, run how, scored how. This isn't the preference-tuning of RLHF (that's the fine-tuning pipeline, learning from human comparisons) — it's the machine that produces a huge stream of task attempts, each honestly scored, for the trainer to learn from.
We'll build the environment farm: sandboxed rollout workers that safely run an agent attempting a task, a scheduler that scales them, a hardened verifier that scores each trajectory (because a gameable one corrupts training), a task bank that keeps difficulty in the productive band, and a trajectory store that closes the loop. The learning-signal math — reward variance and verifier precision — is the RL Environments Engineering handbook; this is the infrastructure that runs it at scale.
What the new pieces do
- Trainerclient
- The training loop that improves the policy from scored trajectories. It needs a large, steady stream of task attempts, each honestly scored — but it should not itself be running or sandboxing agent code.
Step 1 · The baseline
Run rollouts inline in the trainer
The naive setup: the trainer, in its own process, runs each task attempt itself — spins up the agent, executes the code the agent generates, scores it, updates. It works for a toy. What breaks the moment the agent generates real code and you need volume?
The trainer runs each rollout inline — executing agent-generated code in its own process. What's the core problem?
Simplicity that runs untrusted, model-generated code in your training process is a liability, not a virtue. One rollout that generates code which loops forever, reads the wrong files, or crashes takes the whole trainer down — and you can't scale rollout throughput without scaling the trainer with it.
Exactly two problems: safety (running arbitrary agent-generated code in-process means a bad rollout can crash, hang, or compromise the trainer) and scale (rollouts and training are coupled, so you can't throw more machines at generating trajectories without touching the trainer). Both are fatal at real scale.
Format is trivial. The real problems are that the trainer runs untrusted agent code in-process (unsafe and unstable) and that rollout throughput is bolted to the trainer (unscalable).
Inline rollouts run untrusted, model-generated code inside the training process — a bad rollout can hang, crash, or compromise the trainer — and they chain rollout throughput to the trainer, so you can't scale the environment independently. Both safety and scale demand pulling rollouts out of the trainer.
Why this piece earns its place
Both of those reasons are already on the table. What the split quietly hands you is a third thing nobody argues for up front: the two halves get to run on different machines, bought differently. A trainer is long-lived, holds optimizer state, and losing it costs a checkpoint reload plus every in-flight batch, so it wants reserved, reliable hardware. A rollout worker is disposable by construction: kill it mid-attempt and you have lost exactly one trajectory out of an enormous stream. That asymmetry is unavailable while the two share a process, and it is where a lot of the farm's bill is decided — the disposable half can sit on preemptible capacity, the irreplaceable half cannot. The second thing you buy is a readable dashboard. Inline, rollout time and training time are one number, so when throughput sags nobody can say which side stalled, and a stack trace does not tell you whether the bug is in your update step or in code the model wrote thirty seconds earlier. Two processes means two error budgets, two sets of logs, and a bottleneck you can point at.
What the new pieces do
- Rollout Workerservice
- Runs one rollout: an agent (loaded with the current policy) attempts a task, executing whatever code and tool calls it generates. Because that code is untrusted and model-generated, each worker is an isolated, resource-capped sandbox.
Step 2 · Isolation
Sandbox every rollout
Pull rollouts into dedicated workers — but those workers run whatever code the agent generates, which is untrusted by definition. What has to be true of a rollout worker before it's safe to run at scale?
Rollout workers execute agent-generated code. What must each worker be?
Speed without isolation is how one malicious or buggy rollout escapes to read another's data, reach the network, or exhaust the host. Running untrusted model-generated code at scale is only safe if each rollout is contained, even at some speed cost.
Shared state between rollouts is exactly what you must prevent — one rollout could poison another's trajectory, and a resource hog could starve its neighbors. Rollouts must be isolated, not shared.
Right: every rollout runs in a sandbox with strict resource caps and no ambient access. The agent can generate and execute whatever code it wants, but that code is boxed — no network, no cross-rollout access, bounded CPU/memory/time. A rollout that misbehaves is contained to itself and can't take down the farm or poison other trajectories.
Each rollout runs in an isolated, resource-capped sandbox: no network, no access to other rollouts' data, bounded CPU, memory, and time. Agent-generated code can do whatever it likes inside the box, but a rogue or runaway rollout is contained to its own worker — the only way running untrusted model-generated code at scale is safe.
Why this piece earns its place
Two consequences of the box that only show up once you have run one. First, no network means every dependency a task needs has to be baked into the sandbox image, which makes that image part of the task definition and something you version with it. Skip that and the same task scores differently this month than last because a package moved underneath you, and you can no longer tell whether the policy improved or the environment drifted — the one comparison the whole farm exists to make. Second, the resource caps are part of the reward function whether you meant them to be. A rollout killed at its CPU or wall-clock limit produces a trajectory that looks, to everything downstream, identical to an honest failure. Score it that way at volume and you are teaching the model to avoid whatever runs long, including the correct approaches that happen to be slow. So a worker has to record why an attempt ended — solved, failed, or terminated by the box — and that distinction has to survive into the stored trajectory, because nothing later can reconstruct it.
Step 3 · Scale
Schedule rollouts across a fleet
One sandboxed worker is safe but slow. Training needs a huge volume of trajectories. What sits between the trainer and the workers, and what does decoupling them buy?
How should rollouts be produced at the volume RL training needs?
That re-couples the trainer to rollout execution — the exact problem you just escaped. The trainer should consume trajectories, not micromanage a fleet of workers; that's the scheduler's job.
Right: the scheduler decouples producing trajectories from learning. It fans rollout jobs out across as many workers as you provision, hands each worker the current policy to run, and collects results — so you scale trajectory throughput by adding workers, without touching the trainer. Rollout generation and training now scale on independent axes.
Sequential rollouts throttle throughput to one-at-a-time, starving the trainer. Rollouts are independent and embarrassingly parallel — the whole point is to run thousands concurrently across a fleet.
A Rollout Scheduler dispatches jobs across a fleet of sandboxed workers and distributes the current policy each worker runs. Rollout throughput now scales by adding workers, independently of the trainer, which simply consumes the resulting trajectories — production and learning on separate axes.
Why this piece earns its place
A rollout is not a web request, and the difference shapes the whole scheduler. Its duration is long and wildly variable — one attempt gives up in seconds, another runs until its time cap — so the tail sets throughput, not the average, and the number to watch is queue depth rather than worker utilization, because a fleet can be fully busy and still starving the trainer. If the trainer wants a group of attempts on the same task, that group finishes when its slowest member does, and the tempting fix is to over-dispatch and cancel whatever is still running once enough have come back. Resist that shape of it: letting the race pick the survivors discards the slow ones systematically, the same bias the resource cap already threatens. Cancel on the task's own wall-clock budget instead — identical for every attempt, and handed back as a recorded termination rather than a gap in the group. Retry is that trap inverted. A rollout lost to infrastructure, a preempted node or a failed image pull, should be retried. A rollout that ended because the agent's own code failed must not be, because that is a result — and re-running it does not repeat the attempt, it samples a fresh one, so tasks whose generated code fails often quietly collect more rollouts than the scheduler budgeted.
What the new pieces do
- Rollout Schedulergateway
- Dispatches rollout jobs across a fleet of workers, so the farm scales rollout throughput independently of the trainer, and distributes the current policy the workers load for each attempt.
Step 4 · Honest scoring
Score trajectories with a sound verifier
A worker finishes a rollout. Now the critical question: did the attempt actually succeed? Everything the model learns rides on this score. What makes a verifier good enough to train on?
A rollout finished. What matters most about how its trajectory is scored?
Speed is nice but secondary to correctness. A fast verifier that can be fooled feeds the trainer false positives — the model learns to satisfy the check, not do the task. A slightly slower, sound verifier beats a fast, gameable one every time.
Generous partial credit toward a gameable check is how reward hacking gets rewarded. What matters is that a success score means the task was actually solved — a verifier that hands out credit for near-misses that don't work teaches the wrong lesson.
Right: verifier soundness is a training-data-integrity problem. If a degenerate solution (print the expected answer, exploit a loose test) scores as success, the model learns that exploit — reward hacking. The verifier must be hardened (hidden tests, checking the process not just a final string) so that a "success" genuinely means the task was solved. Precision — the fraction of rewarded trajectories that are actually correct — is the number to protect.
The Verifier Service must be SOUND — hardened with hidden tests and process checks — because every false positive is a lie the model trains on. A gameable verifier makes the model learn the exploit instead of the skill (reward hacking); protecting verifier precision (rewarded trajectories that are genuinely correct) is protecting the integrity of the training data itself.
Why this piece earns its place
Where the verifier runs is an architectural decision the diagram makes and nobody states out loud. It is a separate service, outside the sandbox, and it has to be: hidden tests are not hidden if they sit on a filesystem the agent's code can read, and a checker running inside the box is being graded by the very thing it is grading. The rollout produces the attempt; something the rollout cannot touch decides what it was worth. Worth naming the opposite error too, since everything above is about false positives. A verifier that is too strict — rejecting a correct solution over output formatting, or because the task was solved by a route the checker did not anticipate — is a lie in the same training data pointing the other way: gradient pushed away from behaviour that genuinely worked, and specifically away from the unanticipated routes you would most want kept. It is the harder error to find, because it never looks like a bug. A false positive eventually surfaces as a model doing something absurd that scores well. A false negative surfaces as tasks going quiet — successes suppressed on exactly the ones the model is just starting to get right, pushed out of the productive band for a reason that has nothing to do with the policy. Both are corrupted training data. Only one of them announces itself.
What the new pieces do
- Verifier Serviceservice
- Scores each completed trajectory — did the attempt actually succeed? Hardened against gaming (hidden tests, process checks) because every false positive is a lie the model learns from.
Step 5 · The curriculum
Serve tasks in the difficulty band
The farm can safely run and honestly score rollouts. But which tasks should it run? Feeding random tasks wastes compute. What makes a task worth attempting for RL?
Which tasks should the Task Bank serve to the rollout workers?
Correct: RL learns from reward variance, which for a pass/fail task peaks when the model solves it about half the time and vanishes when it always or never succeeds. A task the model always solves (or never solves) produces no gradient — every attempt gets the same reward. The task bank should serve tasks in that middle band, and shift them as the model improves (a curriculum), so compute is spent where learning actually happens.
Tasks the model never solves produce all-failure trajectories with zero variance — no signal to learn from. Maximal difficulty teaches nothing until the model can occasionally succeed; the productive band is intermediate, not maximal.
Tasks the model always solves produce all-success trajectories with zero variance — equally useless. High reward with no spread means no gradient. Learning needs tasks the model solves SOMETIMES.
The Task Bank serves tasks in the productive difficulty band — solved sometimes, failed sometimes — because the learning signal is the reward variance, which vanishes when a task is always or never solved. As the model improves, tasks that used to sit in the band drift out, so the bank acts as a curriculum, continuously supplying tasks where learning actually happens.
Why this piece earns its place
The awkward part is that the band is a measurement of the current policy, not a property of a task, and the measurement decays with every update. So the bank needs a live pass-rate estimate per task, which means the component at the end of the loop feeds the component at the front — whether that arrives as an aggregate over the trajectory store, a counter the verifier emits as it scores, or a tally the scheduler keeps as results return. The store is the natural home because it already holds the history; what matters is that one component owns the number. It is not free either: the only way to learn how hard a task is for this policy is to spend rollouts finding out, so budget for measuring, not only for training. The constraint that actually bites, though, is supply. A farm running well burns through its usable tasks faster than it burns through compute, so authoring or generating tasks becomes the ceiling long before hardware does — a staffing decision someone had to make months earlier. And hold a slice back the workers never see. Keep it out of the store too, because a held-out task that lands there and gets replayed is contaminated as thoroughly as one that was trained on.
What the new pieces do
- Task Bankservice
- Supplies the tasks workers attempt, keeping them in the productive difficulty band — hard enough to fail sometimes, easy enough to succeed sometimes — because a task always-solved or never-solved teaches nothing.
Step 6 · Close the loop
Store trajectories and feed the trainer
Rollouts are run, sandboxed, and honestly scored on band-appropriate tasks. The last piece: get those scored trajectories back to the trainer. What has to be captured, and why store rather than stream straight in?
What should happen to each scored trajectory?
The trajectory — the task, the actions taken, the outcome, the reward — is the training data. A bare reward number with no record of what the agent did can't drive a policy update or be re-analyzed. You need the full trajectory, stored.
Right: each trajectory is stored with everything the trainer needs — the task, the sequence of actions, the outcome, the reward — and flows back to the trainer to update the policy. Storing (rather than only streaming) means trajectories can be batched, replayed, re-scored if the verifier improves, and inspected when something looks wrong. That closes the loop: environment produces scored trajectories, trainer updates the policy, workers run the new policy.
Trajectories are training data, not production output — they include failures and exploited attempts. They flow to the trainer to improve the policy, not to end users.
The Trajectory Store keeps each full scored trajectory — task, actions, outcome, reward — and feeds it to the trainer, closing the loop from environment to policy update. Storing rather than only streaming lets trajectories be batched, replayed, re-scored if the verifier improves, and inspected when a result looks wrong — the environment produces, the trainer learns, the workers run the improved policy.
Why this piece earns its place
Two components stand on both sides of the split: the scheduler carries policy out to the fleet, the store carries trajectories back. The store is the one whose schema is a contract rather than a table. Adding a field is cheap. Changing what an existing one means is not — redefine reward and every record already written silently means something else, so a batch that mixes old and new trajectories is training on two different definitions of success with nothing to tell them apart. Stamp each record with the policy and verifier versions that produced it and you can at least filter; leave that out and the damage is unrecoverable after the fact. The second thing to design for is that the two sides read and write in opposite shapes. The farm appends continuously, at the fleet's rate, in whatever order attempts happen to finish. The trainer wants shuffled random access across a large window, fast enough to keep expensive accelerators fed. Choose one engine that does both adequately and the store quietly becomes the thing idling the costly half of the system.
What the new pieces do
- Trajectory Storedata
- Stores every scored trajectory (task, actions, outcome, reward) and feeds them back to the trainer, closing the loop from environment to policy update.
The payoff
A training engine, not just a scorer
Put it together: the trainer gets a steady, scalable stream of honestly-scored trajectories on band-appropriate tasks, produced by sandboxed workers that safely run untrusted agent code — and a reward-hacking exploit or a rogue rollout is caught, not learned.
That's the RL environment farm. Sandboxed rollout workers, a scheduler for scale, a sound verifier that protects training-data integrity, a difficulty curriculum, and a trajectory store that closes the loop — the environment-and-verifier infrastructure where the real leverage in modern RL sits.
Everything you assembled, in order
- Rollouts are pulled out of the trainer: the farm PRODUCES scored trajectories, the trainer CONSUMES them — separating environment from learning.
- Every rollout runs in an isolated, resource-capped sandbox, because RL on code/tool tasks means executing untrusted model-generated code.
- A scheduler fans rollouts across a fleet and distributes the current policy, so trajectory throughput scales independently of the trainer.
- The verifier must be SOUND — hardened against gaming — because every false positive is a poisoned lesson the model learns (reward hacking).
- The task bank serves the productive difficulty band (a curriculum), and the trajectory store closes the loop back to the trainer.