The big idea

The environment is the training signal

The reasoning models everyone celebrates are trained by RL — but the algorithm gets too much credit. What actually teaches the model is the environment: which tasks it attempts, run how, scored how. This isn't the preference-tuning of RLHF (that's the fine-tuning pipeline, learning from human comparisons) — it's the machine that produces a huge stream of task attempts, each honestly scored, for the trainer to learn from.

We'll build the environment farm: sandboxed rollout workers that safely run an agent attempting a task, a scheduler that scales them, a hardened verifier that scores each trajectory (because a gameable one corrupts training), a task bank that keeps difficulty in the productive band, and a trajectory store that closes the loop. The learning-signal math — reward variance and verifier precision — is the RL Environments Engineering handbook; this is the infrastructure that runs it at scale.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end let an agent game the reward and try to break out of its sandbox to see the farm hold. Hit Begin.

Step 1 · The baseline

Run rollouts inline in the trainer

The naive setup: the trainer, in its own process, runs each task attempt itself — spins up the agent, executes the code the agent generates, scores it, updates. It works for a toy. What breaks the moment the agent generates real code and you need volume?

Design decision: The trainer runs each rollout inline — executing agent-generated code in its own process. What's the core problem?

The call: The trainer executes untrusted, model-generated code in its own process (a safety and stability disaster), and rollout throughput is chained to the trainer, so you can't scale the environment independently of training. — Exactly two problems: safety (running arbitrary agent-generated code in-process means a bad rollout can crash, hang, or compromise the trainer) and scale (rollouts and training are coupled, so you can't throw more machines at generating trajectories without touching the trainer). Both are fatal at real scale.

Inline rollouts run untrusted, model-generated code inside the training process — a bad rollout can hang, crash, or compromise the trainer — and they chain rollout throughput to the trainer, so you can't scale the environment independently. Both safety and scale demand pulling rollouts out of the trainer.

Separate producing trajectories from learning from them: The whole design splits two concerns the baseline conflates: generating scored trajectories (the farm) and updating the policy from them (the trainer). Everything below builds the farm as a safe, scalable producer the trainer just consumes from.

Step 2 · Isolation

Sandbox every rollout

Pull rollouts into dedicated workers — but those workers run whatever code the agent generates, which is untrusted by definition. What has to be true of a rollout worker before it's safe to run at scale?

Design decision: Rollout workers execute agent-generated code. What must each worker be?

The call: An isolated, resource-capped sandbox — so agent-generated code can't reach the network, read another rollout's data, or consume unbounded CPU/memory, and a rogue rollout is contained to its own worker. — Right: every rollout runs in a sandbox with strict resource caps and no ambient access. The agent can generate and execute whatever code it wants, but that code is boxed — no network, no cross-rollout access, bounded CPU/memory/time. A rollout that misbehaves is contained to itself and can't take down the farm or poison other trajectories.

Each rollout runs in an isolated, resource-capped sandbox: no network, no access to other rollouts' data, bounded CPU, memory, and time. Agent-generated code can do whatever it likes inside the box, but a rogue or runaway rollout is contained to its own worker — the only way running untrusted model-generated code at scale is safe.

Untrusted code needs a box: RL on coding and tool-use tasks means executing code the model wrote — inherently untrusted. Sandboxing every rollout is the non-negotiable that makes the whole farm safe to point at real, code-executing tasks.

Step 3 · Scale

Schedule rollouts across a fleet

One sandboxed worker is safe but slow. Training needs a huge volume of trajectories. What sits between the trainer and the workers, and what does decoupling them buy?

Design decision: How should rollouts be produced at the volume RL training needs?

The call: A scheduler dispatches rollout jobs across a fleet of sandboxed workers and distributes the current policy — so rollout throughput scales independently of the trainer, and the trainer just consumes the results. — Right: the scheduler decouples producing trajectories from learning. It fans rollout jobs out across as many workers as you provision, hands each worker the current policy to run, and collects results — so you scale trajectory throughput by adding workers, without touching the trainer. Rollout generation and training now scale on independent axes.

A Rollout Scheduler dispatches jobs across a fleet of sandboxed workers and distributes the current policy each worker runs. Rollout throughput now scales by adding workers, independently of the trainer, which simply consumes the resulting trajectories — production and learning on separate axes.

Fan out the embarrassingly parallel part: Rollouts are independent, so they parallelize trivially. The scheduler turns that into throughput: the trainer's appetite for trajectories is fed by however many workers you're willing to run, with no change to the training loop.

Step 4 · Honest scoring

Score trajectories with a sound verifier

A worker finishes a rollout. Now the critical question: did the attempt actually succeed? Everything the model learns rides on this score. What makes a verifier good enough to train on?

Design decision: A rollout finished. What matters most about how its trajectory is scored?

The call: That the verifier is SOUND — hardened against gaming with hidden tests and process checks — because every false positive is a lie the model trains on, and a gameable verifier makes the model learn the exploit instead of the skill. — Right: verifier soundness is a training-data-integrity problem. If a degenerate solution (print the expected answer, exploit a loose test) scores as success, the model learns that exploit — reward hacking. The verifier must be hardened (hidden tests, checking the process not just a final string) so that a "success" genuinely means the task was solved. Precision — the fraction of rewarded trajectories that are actually correct — is the number to protect.

The Verifier Service must be SOUND — hardened with hidden tests and process checks — because every false positive is a lie the model trains on. A gameable verifier makes the model learn the exploit instead of the skill (reward hacking); protecting verifier precision (rewarded trajectories that are genuinely correct) is protecting the integrity of the training data itself.

The verifier writes the training data: What the verifier rewards is literally what the model learns. A hole in the verifier isn't a bug in a check — it's a poisoned lesson, repeated across millions of trajectories. Hardening it is the highest-leverage correctness work in the farm.

Step 5 · The curriculum

Serve tasks in the difficulty band

The farm can safely run and honestly score rollouts. But which tasks should it run? Feeding random tasks wastes compute. What makes a task worth attempting for RL?

Design decision: Which tasks should the Task Bank serve to the rollout workers?

The call: Tasks in the productive difficulty band — hard enough that the model fails sometimes, easy enough that it succeeds sometimes — because the learning signal comes from the spread between successes and failures. — Correct: RL learns from reward variance, which for a pass/fail task peaks when the model solves it about half the time and vanishes when it always or never succeeds. A task the model always solves (or never solves) produces no gradient — every attempt gets the same reward. The task bank should serve tasks in that middle band, and shift them as the model improves (a curriculum), so compute is spent where learning actually happens.

The Task Bank serves tasks in the productive difficulty band — solved sometimes, failed sometimes — because the learning signal is the reward variance, which vanishes when a task is always or never solved. As the model improves, tasks that used to sit in the band drift out, so the bank acts as a curriculum, continuously supplying tasks where learning actually happens.

Learning lives in the middle: Compute spent on tasks the model always or never solves is wasted — they produce no gradient. The task bank's job is to keep the farm attempting tasks in the band where success is uncertain, and to move the band as the model gets better.

Step 6 · Close the loop

Store trajectories and feed the trainer

Rollouts are run, sandboxed, and honestly scored on band-appropriate tasks. The last piece: get those scored trajectories back to the trainer. What has to be captured, and why store rather than stream straight in?

Design decision: What should happen to each scored trajectory?

The call: Store the full scored trajectory (task, actions, outcome, reward) and feed it to the trainer — closing the loop from environment to policy update, and giving you a re-usable, inspectable record. — Right: each trajectory is stored with everything the trainer needs — the task, the sequence of actions, the outcome, the reward — and flows back to the trainer to update the policy. Storing (rather than only streaming) means trajectories can be batched, replayed, re-scored if the verifier improves, and inspected when something looks wrong. That closes the loop: environment produces scored trajectories, trainer updates the policy, workers run the new policy.

The Trajectory Store keeps each full scored trajectory — task, actions, outcome, reward — and feeds it to the trainer, closing the loop from environment to policy update. Storing rather than only streaming lets trajectories be batched, replayed, re-scored if the verifier improves, and inspected when a result looks wrong — the environment produces, the trainer learns, the workers run the improved policy.

The trajectory is the data: Everything upstream exists to produce this: a stored, honestly-scored record of an attempt. Closing the loop back to the trainer — and keeping the trajectories inspectable — is what makes the farm a training engine rather than a one-shot scorer.

The payoff

A training engine, not just a scorer

Put it together: the trainer gets a steady, scalable stream of honestly-scored trajectories on band-appropriate tasks, produced by sandboxed workers that safely run untrusted agent code — and a reward-hacking exploit or a rogue rollout is caught, not learned.

That's the RL environment farm. Sandboxed rollout workers, a scheduler for scale, a sound verifier that protects training-data integrity, a difficulty curriculum, and a trajectory store that closes the loop — the environment-and-verifier infrastructure where the real leverage in modern RL sits.

• Rollouts are pulled out of the trainer: the farm PRODUCES scored trajectories, the trainer CONSUMES them — separating environment from learning.
• Every rollout runs in an isolated, resource-capped sandbox, because RL on code/tool tasks means executing untrusted model-generated code.
• A scheduler fans rollouts across a fleet and distributes the current policy, so trajectory throughput scales independently of the trainer.
• The verifier must be SOUND — hardened against gaming — because every false positive is a poisoned lesson the model learns (reward hacking).
• The task bank serves the productive difficulty band (a curriculum), and the trajectory store closes the loop back to the trainer.