Reflexion,
explained.

Rungs 1 and 2 get one attempt. If the agent fails, it fails; retrying it gives you a fresh sample from the same distribution, which fails in much the same way. The obvious fix is to learn from the failure, and the obvious way to learn is a gradient step — expensive, slow, and impossible if you do not own the weights.

The change is to learn in words instead of weights. After a failed attempt, the agent is shown the trajectory and asked to write a reflection on why it failed — a short, specific diagnosis, not a score. That reflection is stored and placed in the context of the next attempt. The policy has changed, in the only sense that matters for behaviour, with no parameters touched.

Stated in the units the ladder compares on: the loop is actor, evaluator, reflector — something acts, something determines that the outcome was bad, and something converts that into an instruction the next attempt can use. The reported result on a code benchmark is in the region of 91%, from repeated attempts where a single attempt is far lower. And the cost is a few extra model calls, which against any form of training is close to nothing.

A ReAct-style agent can act, observe, and adapt within one attempt. But let it fail the whole task and try again, and something embarrassing happens: it repeats the same mistake. Nothing carried over. The standard fixes were both bad — fine-tune on failures (expensive, impossible via API, and one task's lessons overwrite another's) or just replay the failed attempt in context (raw trajectories help some, but a short written lesson distilled from the failure is a stronger and more compact signal).

What's missing is what humans do between attempts: compress the failure into a lesson. Not "here are the 4,000 tokens of my failed attempt" but "I forgot to handle the empty-list case."

Reflexion splits the retry loop into three roles. The Actor attempts the task — writing code, navigating a household simulator, searching for answers. The Evaluator grades the attempt: unit tests for code, task success for environments, an LLM judge where nothing crisper exists. And the Self-Reflection model turns the graded failure into a short, specific verbal lesson.

The lessons accumulate in an episodic memory buffer that's prepended to every new attempt. Attempt three doesn't just know the task — it knows exactly how attempts one and two died.

Classic RL compresses feedback into a single number — reward 0. Everything about why is discarded, and the algorithm spends thousands of episodes rediscovering it through trial and error. A verbal reflection keeps the why: which assumption was wrong, which edge case was missed, what to try instead. It's credit assignment in prose — precise, immediate, and executable by the very next attempt.

The whole loop is one line of state that survives between attempts — the memory of what went wrong:

Delete that memory arrow and every retry is attempt one again. The runnable version below shows it fix the boundary bug on attempt two — and a memoryless retry failing forever.

Reflexion learns nothing durable — lessons live in a context buffer, bounded by the window, gone when the episode ends (unless you persist them into real memory). It leans on the Evaluator: crisp signals like unit tests make it soar; fuzzy self-evaluation can reinforce confident nonsense. And reflection quality is capped by the model's own diagnostic ability — an agent that misunderstands why it failed writes itself bad advice and then dutifully follows it.

It's also, quietly, a test-time method: multiple attempts cost multiple rollouts. Like self-consistency and ToT, it buys capability with inference compute.

Reflexion named and validated the pattern that now lives inside nearly every serious agent: try → check → diagnose in words → retry with the diagnosis. Every coding agent that reads its own test failures and patches, every harness with a critique-then-revise step, every eval loop that feeds errors back as guidance — that's the Reflexion loop, productionized.

Its deeper claim aged even better: a model's language channel is a legitimate substrate for learning. That idea threads through self-critique training, agent post-mortems written to memory, and the whole self-improvement research line. Weights are one place to store lessons; words, it turns out, are another.

Reflexion is a loop around an existing agent, built from three prompted models and a short memory. Nothing is trained; every component is a language-model call.

1. Choose the actor. The actor M_a is whatever agent suits the task: ReAct for the ALFWorld household games and for HotpotQA with a Wikipedia API, chain of thought for HotpotQA with the supporting paragraphs supplied, and a code writer for programming. Its policy is π_θ(a_i | s_i) with θ = {M_a, mem}: the frozen model plus whatever is in memory.
2. Run one trial. The actor produces a trajectory of actions and observations, conditioned on the task and on the memory.
3. Score it. The evaluator M_e returns a sparse signal, usually pass or fail. On HotpotQA it is exact match against the answer. In ALFWorld, where the environment only reports completion, it is either an LLM classifier or a hand-written heuristic that flags failure when the agent repeats the same action and gets the same response more than three times, or takes more than 30 actions. For code it is a test suite the agent writes itself: chain-of-thought prompting proposes tests, any that do not parse into a valid syntax tree are dropped, and up to six are sampled.
4. Reflect. The self-reflection model M_sr reads the signal, the trajectory and the current memory and writes a short diagnosis with a plan: in one ALFWorld example, that it should have found the desk lamp first and then the mug, not the other way round.
5. Store it, within a cap. The reflection is appended to an episodic memory that keeps only the last Ω entries, usually 1 to 3 so the prompt fits the context window: three for ALFWorld and HotpotQA, one for programming.
6. Retry. The next trial starts with the memory in the prompt. The loop stops when the evaluator passes or the trial budget runs out. ALFWorld ran for 12 trials; on HotpotQA an agent retried a question until it failed three times in a row.
7. Submit once. For code, only the final program is scored against the benchmark's hidden tests. All the iteration happens against the agent's own tests, which is what keeps the reported number a pass@1.

The baselines are the same agents without the reflection step. In ALFWorld, a baseline run that trips the heuristic simply resets and starts a new trial with nothing learned. On HotpotQA, sampling at temperature 0.7, the baselines solved none of their first-trial failures on any later trial.

The models differ by task: GPT-4 for the programming results, a GPT-3 model with ReAct's two few-shot trajectories for ALFWorld, and text-davinci-003, gpt-3.5-turbo and GPT-4 compared on 100 HotpotQA questions. HotpotQA prompts used 6 examples for chain of thought, 2 for ReAct and 2 for self-reflection.

• θ — the parameters of the policy. Reflexion's move is that one of them is text: updating mem changes behavior without touching Ma.
• rt — the evaluator's reward, usually binary: exact match, task success, or all self-written tests passing.
• srt — the verbal reinforcement: a few sentences naming what went wrong and what to do instead.
• Ω — the memory cap, 1 to 3 reflections.

The extremes map onto the failure modes. With Ω = 0 every retry is a fresh sample from the same policy, and on HotpotQA such retries fixed nothing. If Me passes a wrong answer, the loop stops early and never reflects, which is how false-positive self-tests hurt MBPP. If Msr cannot diagnose its own errors, the memory fills with useless advice: with the smaller starchat-beta model, HumanEval pass@1 stayed at 0.26 with or without Reflexion.

The programming runs use GPT-4 with self-generated tests and report pass@1; each baseline is the same model's single attempt.

Plain GPT-4 at 80.1 was the state of the art Reflexion had to beat; the best earlier method the authors list, CodeT with GPT-3.5, scored 65.8. LeetcodeHardGym is the authors' own set of 40 hard Leetcode problems released after GPT-4's pretraining cutoff, to avoid contamination. On ALFWorld the ReAct-only agent settled at a 22 percent hallucination rate with no sign of recovering; a common baseline error was believing it held an item it did not.

Two ablations isolate what does the work. On the 50 hardest HumanEval Rust problems the full method scored 0.68 against a 0.60 baseline. Keeping the self-written tests but dropping reflection stayed at 0.60. Keeping reflection but dropping the tests fell to 0.52, below the baseline, because an agent that cannot tell when its code is correct never stops early and keeps editing working programs. On HotpotQA with the supporting paragraphs given, reflection added 8 points over episodic memory alone, which simply replays the last trajectory.

The misses are as informative. MBPP Python got worse, 80.1 to 77.1, because the self-written tests were unreliable there: 16.3 percent of submissions that passed the agent's own tests failed the hidden ones, against 1.4 percent on HumanEval Python, and a false positive ends the loop on wrong code. On WebShop, two-shot ReAct plus Reflexion showed no improvement after four trials across 100 environments, and its reflections were unhelpful, which the authors read as a limit on tasks needing diverse exploration. And the method needs a strong model: with starchat-beta, HumanEval pass@1 was 0.26 with and without reflection.

The loop needs an evaluator, and that is where it is strong or hopeless with nothing in between. Where failure is detected by running tests — code, as in the headline result — the signal is reliable and the method works well. Where the agent must judge its own success, the loop inherits a known weakness: models are poor at recognising their own errors, and an agent that believes it succeeded will not reflect at all.

The second failure mode is that a reflection can be wrong. A confident, incorrect diagnosis of the failure is now in the context, actively steering the next attempt away from the answer, and nothing removes it. Verbal lessons accumulate without being validated, which is a different risk profile from a gradient step that at least moved in the direction of a measured loss.

How it held up: retry-with-reflection is in every agent framework, because it is the cheapest self-correction available and it needs no infrastructure. The framing — natural language as the medium of policy improvement — is the durable idea, and rung 4's answer to the wrong-reflection problem is to store something that can be executed and tested rather than merely read.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Give the model a tool, then a loop, then a memory, then a world. Ends on the 2026 paper measuring how badly the whole stack degrades over many turns.

1. 1ReAct2023
2. 2Toolformer2023
3. 3Reflexion2023
4. 4Voyager2023
5. 5Generative Agents2023
6. 6MemGPT2023
7. 7LLMs Get Lost in Multi-Turn Conversation2026

All 110 papers, in 19 ladders →