Here is a strange, useful fact about large language models: you can often make one much better at a problem without changing the model at all — just by changing how you ask. A model generates text one token at a time, and each token is a fixed, bounded amount of computation. So when you ask a multi-step question and demand only the final answer, you’re forcing the entire chain of reasoning to be compressed into the handful of tokens before the answer — and under that pressure the model tends to grab a plausible shortcut (anchor on a salient number, do one operation instead of three) and gets it wrong. Chain-of-thought prompting — literally “let’s think step by step” — removes that pressure. It invites the model to spend intermediate tokens writing out the sub-steps, and each individual step is small enough that the model reliably gets it right. Those written-out steps then become part of the context the next step reads, so the model is effectively computing on paper instead of in its head. Run the same problems in both modes below: watch the direct answer take the tempting shortcut and miss, and the step-by-step chain arrive correct.
one token = bounded compute · direct answer crams every step into one · chain-of-thought spends tokens per step
chain-of-thought
Prompting the model to write out intermediate reasoning steps before the final answer.
direct answering
Asking only for the final answer, forcing the whole computation into the tokens before it.
token = compute
Each generated token is a fixed slice of computation; more tokens means more room to work.
decomposition
Breaking a hard problem into small steps, each reliable, so errors don’t compound into a wrong blurt.
reason.js — spend tokens, get it right
Ready
Pick a problem, then answer it two ways. Answer directly forces every step into one blurt — watch it take a tempting shortcut and miss. Think step by step reveals the chain one step at a time, each small and correct, landing on the right answer.
—
mode
—
direct answer
—
chain answer
How it works
The reason chain-of-thought works isn’t magic; it follows from how these models compute. A transformer does a fixed amount of work per token — one forward pass — and then commits to a token and moves on. That means the number of sequential reasoning steps a model can perform before it must produce the answer is essentially bounded by how many tokens it generates. Ask for the answer immediately and you cap the available computation at roughly one step’s worth, so any problem that genuinely needs several dependent steps gets squeezed, and the model falls back on pattern-matched shortcuts that are right on easy cases and wrong on the ones that need actual work. Chain-of-thought lifts the cap: by generating the intermediate steps as text, the model (a) gets more forward passes to compute with, and (b) writes each partial result into the context, so later steps can read earlier conclusions instead of having to hold everything implicitly. This is why the technique reliably helps on arithmetic, multi-hop questions, logic, and code, and why it barely helps (or can even hurt) on simple one-step tasks where the shortcut was already correct — there was nothing to decompose. It also explains several relatives of the idea: self-consistency (sample several independent chains and take the majority answer, since correct reasoning tends to converge while errors scatter), least-to-most prompting (solve easier sub-problems first and feed them forward), and the modern “reasoning” models that are trained to produce long internal chains before answering — all of them are, at bottom, ways of buying the model more room to think in tokens. The practical caveats are worth knowing: a written chain is not a guaranteed-faithful account of the model’s internal process, and a confident-looking chain can still contain a wrong step, so chains help accuracy but don’t replace verification.
1
One token, one bounded step
The model computes a fixed amount per token. A multi-step problem needs several dependent computations, but a direct answer gives it almost no room to perform them.
2
Direct → tempting shortcut
Under that pressure the model grabs a plausible shortcut — anchoring on a prominent number or doing one operation instead of the full chain — and produces a confident but wrong answer.
3
Step by step → room to compute
Chain-of-thought lets the model write each sub-step as tokens. Each step is small enough to get right, and each written result becomes context the next step can read — computing on paper, not in its head.
✓
The chain lands correct
The sequence of reliable small steps reaches the right answer. This is why chain-of-thought helps on multi-step tasks (and barely helps on one-step ones) — and why self-consistency and reasoning models extend the same idea.
Per token
fixed compute
Direct answer
one step of room
Chain-of-thought
a step per line
Helps most on
multi-step problems
The code
# Same model, same problem — two prompts# Direct: all reasoning crammed before the answer token
prompt = problem + "\nAnswer:" # -> often a shortcut, wrong# Chain-of-thought: invite intermediate tokens
prompt = problem + "\nLet's think step by step:"
# -> step 1 ... step 2 ... step 3 ... therefore ANSWER (reliable)# Self-consistency: sample several chains, take the majority answer
answers = [sample(cot_prompt) for _ in range(k)]
final = majority_vote(answers) # correct chains converge
Quick check
1. Why does forcing a direct answer to a multi-step problem often produce a wrong result?
A model does a bounded amount of work per token. Asking only for the final answer caps the available computation at about one step, so a problem needing several dependent steps gets squeezed and the model falls back on a plausible but wrong shortcut.
2. How does chain-of-thought prompting help the model compute?
Writing out the steps gives the model more forward passes to compute with and records each partial result in the context, so later steps read earlier conclusions instead of holding everything implicitly. It’s effectively computing on paper — each small step is reliable.
3. On which kind of task does chain-of-thought help the least?
Chain-of-thought helps when a problem genuinely needs several dependent steps. On simple one-step tasks the shortcut is already right, so decomposing adds little and can even hurt. The benefit scales with how much real sequential computation the task requires.
FAQ
What is chain-of-thought prompting?
Asking a model to write out its intermediate reasoning steps before the final answer, often via “let’s think step by step.” Since a model does a bounded amount of computation per token, generating the steps as text gives it more room to work and lets each step build on the previous written result. This substantially improves accuracy on problems needing several dependent steps — arithmetic, multi-hop questions, logic, code — versus asking for the answer directly.
Why does thinking step by step improve accuracy?
It relaxes a computational bottleneck. A transformer does one forward pass per token, so the sequential reasoning it can do before answering is roughly limited by how many tokens it generates. An immediate answer caps that at about one step — not enough for multi-step problems, so the model shortcuts. Writing steps provides more computation and externalizes partial results into context, so later steps reference earlier conclusions. Each small step is reliable, so the chain reaches the right answer far more often.
What is self-consistency and how does it relate to chain-of-thought?
Self-consistency samples several independent chains of thought for the same problem and takes the majority answer, rather than trusting one chain. The intuition: correct reasoning paths tend to converge on the same answer while errors scatter, so voting favors the correct one. It reliably improves on plain chain-of-thought at the cost of more compute (several samples). It’s one of a family — with least-to-most prompting and trained reasoning models — that give the model more room and attempts to reason.
Is a model’s chain of thought a faithful account of how it actually reasons?
Not necessarily. The chain is generated text; it usually helps the model reach a better answer but isn’t guaranteed to truthfully record the internal computation. Models can reach an answer for reasons not reflected in their stated steps, and a fluent chain can still contain a wrong step. So treat chain-of-thought as an accuracy-improving tool with some insight, not a verified explanation — in high-stakes settings the reasoning and answer still need independent checking, and methods like self-consistency or verifier models catch plausible-but-wrong chains.