Chain-of-Thought
prompting, explained.

Rung 1, so the comparison is against the few-shot prompting it modifies, and the modification is almost absurdly small. Standard few-shot prompting shows the model a handful of question-and-answer pairs and then a new question. Chain of thought shows it question, worked reasoning, answer — and then the new question. Nothing else changes. No training, no gradient, no parameters, no architecture.

Why that should matter at all is the interesting part. A model answering immediately has to produce the answer in a single forward pass, so the amount of computation available is fixed by the architecture regardless of how hard the question is. Generating intermediate steps lets the model take more forward passes on a harder problem (though the paper's dots-only ablation suggests the extra passes alone are not what helps), and lets each step condition on the written result of the last — externalising working memory into the context window, where it is exact, instead of holding it in activations, where it is not.

The second finding is the one that made the paper famous and is worth carrying up the ladder carefully. The benefit is not uniform across model sizes: below a certain scale, showing the reasoning does not help and can actively hurt, because a smaller model produces fluent chains that are wrong and then follows them. The gain appears at scale and appears sharply, which is where the vocabulary of emergent abilities attached to this result.

By 2022, big language models were fluent and knowledgeable, yet strangely bad at problems a careful ten-year-old could solve — multi-step arithmetic, word problems, chains of logic. A model prompted to answer straight away had to squeeze every step of the calculation into a single forward pass and blurt the result. On anything needing more than one step, it stumbled.

The failure wasn't knowledge. It was being forced to leap from question to answer with no room to work in between.

The authors changed the prompt, not the model. In the few examples you show a model before asking your real question — the few-shot exemplars — they didn't just include the answer. They included the reasoning that leads to it, spelled out step by step. That trail of intermediate steps is the chain of thought.

Primed with a couple of examples of what good reasoning looks like, the model imitates the format: for your new question, it writes out its own steps before committing to an answer — giving itself the scratch paper it never had.

Nothing about the model changed between these two panes — only whether it was allowed to think out loud. Breaking one hard leap into a handful of easy steps, each building on the last, is the entire trick.

There's a simple reason this compounds. If each reasoning step is right with probability p, the whole answer needs all k steps to land — p^^k. Answering in one shot means each hidden step is less reliable (no room to work), so its p is lower, and p^^k falls off a cliff:

The more steps a problem needs, the wider the gap — which is exactly where the paper found CoT helps most. The runnable version below makes that curve.

The paper's most striking finding: chain-of-thought is an emergent ability. On small models it barely helps, and sometimes hurts — they produce reasoning that sounds right but doesn't hold together, and the wrong steps drag the answer down with them.

Then, somewhere past roughly 100 billion parameters, the curve bends sharply upward. Suddenly the reasoning is coherent, the steps are valid, and accuracy jumps. Reasoning wasn't something you could prompt into a small model; it appeared, almost like a phase change, only once the model was big enough.

A closely related follow-up found you often don't even need worked examples. Append five words — "Let's think step by step" — to a question, and a large model starts generating its own reasoning chain unprompted. Much of the benefit of chain-of-thought, unlocked by a single sentence.

Chain-of-thought reframed inference itself: a model that can spend more steps "thinking" before answering is more capable than one forced to reply instantly. That single insight seeded everything that followed — self-consistency (sample many chains, take the majority vote), tree-of-thought, and the modern reasoning models that are explicitly trained to generate long internal chains and spend extra compute at inference on hard problems.

When today's models "think" before they answer, they're standing on this paper. It's the moment the field learned that how you let a model compute can matter as much as how big it is.

Chain-of-thought prompting comes from Jason Wei and colleagues at Google Research's Brain team. Nothing in the model changes. The whole method is how the prompt is built and how the answer is read back out, and the paper's contribution is doing that carefully across many models and tasks and then taking it apart with ablations.

1. Write the exemplars by hand. For the math word-problem benchmarks the authors wrote one set of eight exemplars, each a question, a short chain of reasoning in plain language, and a closing line stating the answer. The same eight were reused for GSM8K, SVAMP, ASDiv and MAWPS. AQuA, which is multiple choice, got four exemplars taken from its training set. The commonsense and symbolic tasks got their own small hand-written sets.
2. Build the baseline from the same exemplars. Standard few-shot prompting uses the identical questions and answers with the chains deleted, so the only difference between the two prompts is the reasoning text.
3. Decode greedily and parse. The exemplars and the test question go in, and the model generates until it stops. Decoding is greedy, so each question gets exactly one chain, and the answer is read from the closing answer line the exemplars taught the model to write.
4. Sweep model families and sizes. The same prompts ran on LaMDA (422M to 137B parameters), GPT-3 (InstructGPT variants from about 350M to 175B), PaLM (8B, 62B and 540B), UL2 20B and Codex. Sweeping size inside each family is what exposes the scale threshold.
5. Cover three kinds of reasoning. Arithmetic on five math benchmarks; commonsense on CSQA, StrategyQA, date and sports understanding from BIG-bench, and SayCan robot planning; and symbolic manipulation, last-letter concatenation and coin flips, with test inputs longer than any exemplar showed.
6. Ablate what the chain contributes. Three variants each isolate one explanation. Equation only: the exemplars show just the equation before the answer. Variable compute only: the exemplars show a run of dots as long as the equation, so the model spends extra tokens that carry no content. Reasoning after the answer: the chain follows the answer, so it cannot have produced it.
7. Test robustness. Chains written by three different annotators, exemplars sampled from GSM8K's training set, and different exemplar orders and counts.

What the ablations rule out

Each ablation matches a cheap explanation of the gain. If it came from translating words into equations, equation-only prompting would match it; on GSM8K it helps little, though it does help on one- and two-step problems. If it came from extra tokens of computation, the dots would match it; they score about like the baseline. If the chain only reminded the model of relevant knowledge, moving it after the answer would keep the gain; it does not. What is left is that the content of the steps, written in order before the answer, is what helps.

Reading the errors

The authors also read chains by hand. For LaMDA 137B on GSM8K, nearly every chain that reached a correct answer was itself correct, with two that got there by coincidence. Among wrong answers, about half the chains were nearly right, spoiled by a calculator slip, a symbol-mapping error or one missing step, and the rest had major semantic or coherence errors. Comparing PaLM 62B with 540B, the larger model fixed a large share of the missing-step and semantic-understanding errors, which is the scale threshold seen one error at a time.

Chain-of-thought prompting changes what the model conditions on, not how it is trained. With k exemplars, each a question x_i, a chain c_i and an answer y_i, the model decodes greedily and the answer is parsed from the end of its output:

• xi ci yi — the hand-written exemplars; delete every c_i and this is standard few-shot prompting
• x — the test question
• ẑt — the token chosen at step t; the argmax is greedy decoding, so one question yields one chain
• parse — reads the number or option from the closing answer line the exemplars taught

The rule commits to a single path: the answer is right only if that one chain is. At one extreme, k = 0 plus a trigger sentence is zero-shot chain of thought. At small model sizes the chain itself goes wrong, and parse faithfully reports the wrong answer it ends on. Replacing the argmax with sampling and voting over many chains is exactly the change the self-consistency paper makes next.

The headline test is GSM8K, grade-school math word problems, with the same eight exemplars used with and without their chains, greedy decoding, and solve rate as the score. The prior best was GPT-3 175B fine-tuned on GSM8K's training set and paired with a separately trained verifier, at 55%.

PaLM 540B with eight exemplars and no training more than tripled its own baseline and passed the fine-tuned system. The pattern held beyond math: on StrategyQA, PaLM 540B with chain of thought reached 75.6% against a prior state of the art of 69.4%, and on sports understanding it reached 95.4%, above the 84% the paper cites for an unaided sports enthusiast.

Where it failed. Below roughly 100B parameters the chains were fluent but illogical, and accuracy did not improve; on some tasks it fell below standard prompting. On the easiest subset of MAWPS, problems needing a single operation, the gain was negative or very small, because there is no chain to write. The three ablations were negative by design: equation only, dots only and reasoning after the answer each recovered little or none of the GSM8K gain. Chains from three different annotators all beat the baseline by a wide margin, though they varied among themselves, which is both the robustness claim and its limit.

The paper is careful about something that its popular summary is not: the generated chain is not a trace of the model's computation. It is text the model produced, which influences what it produces next. Whether it describes the process that actually determined the answer is a separate empirical question, and later work on faithfulness found the two can come apart — models will produce a chain that does not reflect the features actually driving their answer, including cases where an answer is steered by something the chain never mentions. Reading a chain of thought as an explanation is therefore a mistake with a specific name, and it matters most where the stakes are highest.

The emergence framing was also contested, usefully. Schaeffer and colleagues argued that apparent sharp jumps are frequently artefacts of discontinuous metrics — score a multi-step task by exact match and smooth underlying improvement shows up as a sudden threshold, because partial credit is unavailable. That critique does not make the practical effect unreal, and it changes what it is evidence of.

How it held up: it is the substrate for everything above it on this ladder and for a large share of the post-training ladder too. The most remarkable thing about it in retrospect is the ratio — a change to the wording of a prompt, requiring no resources at all, that unlocked a capability people had assumed needed a different kind of model. That ratio is also why the rest of this ladder exists: once it was clear the capability was latent, the question became how much compute it was worth spending to extract it.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

It began as a prompting trick and ended as a training objective. Read in order, the progression from "let's think step by step" to a model trained to.

1. 1Chain-of-Thought Prompting2022
2. 2Self-Consistency2022
3. 3Tree of Thoughts2023
4. 4OpenAI o12024
5. 5DeepSeek-R12025

All 110 papers, in 19 ladders →