RL environments
engineering.

In reinforcement learning from verifiable rewards, a model attempts a task, a verifier scores the attempt, and the score nudges the policy. The verifiable-rewards handbook covers why this works and how it fails — reward hacking, verifier gaming. This one is the other half: how to build the environment so the signal is worth learning from.

And the environment is everything the model learns from. Two levers dominate: the tasks (which problems, at what difficulty) and the verifier (how correctness is judged). Both have a measurable failure mode that no amount of algorithmic cleverness fixes — and both are quantifiable, which is what the rest of this handbook does.

Policy-gradient RL learns by comparing better attempts to worse ones. So the signal depends on there being a spread. For a pass/fail task with success probability p, the reward variance — a proxy for the gradient magnitude — is p(1−p). That's a hill: zero at the ends, maximal in the middle.

A task the model always solves (p=1) or never solves (p=0) has zero variance — every attempt gets the same reward, so there's nothing to learn from. The productive tasks are the ones it solves sometimes.

This is why a curriculum matters: as the model improves, tasks that used to sit at p≈0.5 drift toward p≈1 and stop teaching. A good environment continuously supplies tasks in the band — hard enough to fail sometimes, easy enough to succeed sometimes. The runnable model below plots the signal hill.

The task sets the difficulty; the verifier sets the truth. And a verifier that can be fooled is worse than useless — it produces false positives, rewarding trajectories that are actually wrong. The model, being an excellent optimizer, learns to satisfy the verifier rather than solve the task. That's reward hacking, and its root is a low-precision verifier.

Every false positive is a lie in the training data. A verifier that rewards 20 correct and 80 lucky-wrong trajectories has precision 0.2 — it's teaching the wrong lesson 80% of the time.

The theory of how models find and exploit these holes — verifier gaming, specification gaming — is in the verifiable-rewards handbook and the DeepSeek-R1 and GRPO breakdowns. The environment-engineering takeaway is simpler: precision is a first-class environment metric. Harden the verifier — exhaustive test cases, hidden tests, format and process checks — until a wrong answer can't pass.

Strip away the frameworks and every RL environment for LLMs is three functions. A task is what the model sees plus what it must not see. A rollout runs the model against the task, possibly over many turns of tool use. A reward grades the finished trajectory. It's the contract Gym made standard for classic RL, reset() then step(action) returning an observation, a reward and whether the episode is over, stretched to text. The observation is a conversation, and an action is a message or a tool call.

A framework-neutral sketch. Real libraries add batching, async rollouts and token bookkeeping, but the shape stays the same.

The sandbox carries the security model, because the policy is now writing code that you execute millions of times. Run every rollout in a fresh, isolated container. Cap CPU, memory and wall-clock time, and keep the network off unless the task needs it. Keep the hidden tests and the reward computation outside the container: a model that can read the tests will learn to read them, and a model that can edit them will learn to edit them. That's verifier soundness again, enforced by isolation rather than by cleverness. (The Sandboxing handbook and Design a Code Execution Sandbox cover the isolation layer.)

The shape scales to real software work. SWE-Gym (December 2024) packages 2,438 real-world Python tasks, each with a repository and a runnable environment, unit tests, and a task written in plain language. Its authors report up to 19% absolute gains in resolve rate on SWE-Bench Verified and Lite from training agents on it.

The difficulty band from section 02 becomes a concrete filter once you can run rollouts. Sample k attempts per task with the current policy, grade them, and look at the pass rate. GRPO-style methods score each attempt against its own group, A_i = (r_i − mean(r)) / std(r), so when all k rewards are equal every advantage is zero and the task contributes nothing but compute.

Run offline, this is difficulty filtering of the task pool. DAPO (ByteDance Seed and Tsinghua AIR, March 2025) does it online and calls it dynamic sampling: "over-sample and filter out prompts with the accuracy equal to 1 and 0," so every prompt in a batch carries gradient and the batch size stays constant. In the paper's ablation on Qwen2.5-32B, dynamic sampling was the last technique added, and it took AIME 2024 accuracy from 42 to 50. Naive GRPO had started at 30.

DAPO also shows what verifier hardening looks like in practice. Competition answers come as expressions and formulas that are hard to parse reliably, so the team had an LLM rewrite each question until its answer was an integer: a question whose answer is (a+√b)/c becomes one that asks for a+b+c. The result, DAPO-Math-17K, is 17K prompts that each have an integer answer, which a rule can check without a formula parser. The reward is just as plain: 1 if the answer is equivalent to the ground truth, −1 otherwise.

Rule-based rewards are a deliberate choice. DeepSeek-R1 paired an accuracy reward (a boxed final answer for math, compiler feedback on predefined test cases for LeetCode problems) with a format reward for putting the reasoning between think tags. Its authors skipped neural reward models for R1-Zero because they "may suffer from reward hacking in the large-scale reinforcement learning process." Much of the work people want trained, though, has no exact checker: a medical answer, a research summary, a support conversation.

The current answer is to make the grader structured. Rubrics as Rewards (July 2025) has an LLM judge score each response against a checklist of rubric criteria, and uses that score as the reward. Against judges that give one Likert score, the best variant reports relative gains of up to 31% on HealthBench and 7% on GPQA-Diamond, and rubric rewards held up better with smaller judges. A rubric also gives partial credit for free, because satisfying four of six criteria scores higher than satisfying two.

Treat a judge the way section 03 treats any verifier. Measure its precision against human labels on a held-out set, re-check it as the policy improves, and watch for answers that score well on the rubric while getting worse to a human reader. The LLM-as-Judge Prompt Builder is a quick way to draft rubric criteria.

The open recipes converge on the same few environment decisions. Use rule-based rewards wherever a rule exists (DeepSeek-R1, DAPO). Reshape tasks until the rule is reliable (DAPO-Math-17K's integer answers). Filter out prompts the policy always or never solves (DAPO's dynamic sampling). Build real execution environments with hidden tests for agentic work (SWE-Gym). Environments are becoming shareable artifacts too: Prime Intellect's open-source verifiers library, "our library for creating environments to train and evaluate LLMs," is built around publishing them to its Environments Hub.

For an engineer entering this space, the mental model is refreshingly concrete: you're not tuning a mysterious learner, you're building a curriculum with a grader. Keep the tasks in the difficulty band, keep the grader honest, and the algorithm mostly takes care of itself. Get either wrong and no algorithm can save you.