Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
Let the model think — the second scaling axis.
For years, better meant bigger: more parameters, more pretraining data. Reasoning models opened a second dial — spend more compute at inference, and a model that thinks can beat a much larger one that answers instantly. Here’s how test-time compute actually works, how o1- and R1-style models are trained to use it, and when it’s worth the latency.
What “test-time compute” means
Every model call spends compute. Train-time compute is what you burned once, up front, to create the weights. Test-time compute (or inference-time compute) is what you spend each time you answer a question. For a normal chat model, that per-answer cost is roughly fixed: it reads the prompt and emits a few hundred tokens.
A reasoning model breaks that assumption on purpose. Given a hard problem, it generates a long internal chain of thought — hundreds or thousands of “thinking” tokens — working the problem out step by step before committing to a final answer. It is deliberately spending more test-time compute to buy accuracy. The bet, borne out on math, coding, and logic benchmarks, is that for hard problems, a few extra seconds of thinking is worth more than a few extra billion parameters.
A base model answers from intuition, like a person blurting the first thing that comes to mind. A reasoning model gets to use scratch paper — and hard problems are exactly the ones where scratch paper matters.
This is why the same underlying model can be dramatically better or worse depending on how much you let it think. Thinking is no longer a fixed property of the model; it’s a budget you allocate per request.
Two scaling axes, not one
The last decade of progress came from scaling training. Test-time compute is a second, largely independent axis — and it has its own returns curve.
| Axis | What you scale | Paid | Best for |
|---|---|---|---|
| Train-time | Parameters, data, pretraining FLOPs | Once, up front | Broad knowledge, fluency, everything a little better |
| Test-time | Thinking tokens, samples, search per query | Every request | Hard, verifiable reasoning — math, code, logic, planning |
The important research result is that these trade off against each other. On many hard tasks, a smaller model given a generous thinking budget matches or beats a much larger model answering in one shot — and the compute-optimal system is a balance of the two, not a maximum of either. If you have a fixed FLOPs budget to serve a workload, part of the art is deciding how much goes into a bigger model versus how much you leave for the model to think at inference.
There’s a reason this axis got attention exactly when it did: pretraining scaling is showing diminishing returns and is bounded by available data and cost. Test-time compute is a fresh lever with steep early returns on the problems that matter for agents and tools — a way to keep improving capability without training an ever-larger base model.
The four ways to spend it
“Think longer” is not one technique — it’s a family. They can be combined, and they trade cost for accuracy differently.
1 · Long chain-of-thought
- One long generation: the model reasons step by step, backtracks, checks itself, then answers.
- This is what o1/o3 and R1 do natively — the “thinking” is learned, not prompted.
- Cost scales with how many thinking tokens it emits (the reasoning budget).
2 · Sampling + self-consistency
- Generate N independent answers at temperature, then take the majority vote.
- Works when the right answer is a stable attractor and wrong answers are scattered.
- Cheap to implement; accuracy rises with N, then plateaus.
3 · Best-of-N with a verifier
- Sample N candidates, then a verifier or reward model scores them and picks the best.
- Far stronger than voting when you can check answers better than generate them.
- Process reward models score each reasoning step, not just the final answer.
4 · Search
- Explore a tree of partial reasoning steps, expanding promising branches (beam / MCTS-style).
- Uses a verifier to guide where to spend the next unit of compute.
- Most powerful and most expensive; shines on problems with checkable sub-steps.
The unifying principle is generate-and-verify: whenever checking a solution is easier than producing one, you can spend test-time compute producing many candidates and let verification find the good one. That gap between generation and verification difficulty is exactly what these methods exploit — and it’s why math and code, where answers are checkable, benefit most.
How reasoning models learn to think
You can bolt sampling and verifiers onto any model. But models like o1/o3 and DeepSeek R1 go further: the long chain-of-thought is trained in, so a single generation reasons well on its own. The dominant recipe is reinforcement learning from verifiable rewards (RLVR).
The idea is elegant. Take problems with checkable answers — math with a known result, code with tests. Let the model generate a long reasoning trace, then reward it purely on whether the final answer is correct, with no human labeling the steps. Optimize that reward, and the model discovers, on its own, that thinking longer, checking its work, and backtracking lead to more correct answers. DeepSeek’s R1-Zero showed this emergent behavior striking: the reasoning length grew during training, and “aha”-style self-correction appeared without ever being explicitly taught.
A learned reward model can be gamed; a unit test cannot. Grounding the reward in something objectively checkable is what lets you crank up RL without the model learning to produce convincing-but-wrong reasoning. That’s the whole subject of the RLVR handbook.
The result is a model whose default mode is to spend test-time compute wisely — often exposed to you as a reasoning-effort setting (low / medium / high) that dials the thinking budget up or down per request.
When it helps — and when it just burns money
Test-time compute is not free capability; it’s a trade of latency and cost for accuracy, and the trade only pays on the right problems.
✓ Worth it
- Competition math, hard coding, multi-step logic and planning.
- Anything with a checkable answer, where generate-and-verify applies.
- High-stakes single answers where being right dominates being fast.
- Agentic tool use, where one bad step compounds.
✗ Wasteful
- Simple lookups, formatting, chit-chat — a base model already nails these.
- Knowledge the model lacks — thinking harder won’t invent facts; retrieval is the fix.
- Latency-sensitive UX where seconds of “thinking” break the experience.
- High-volume cheap tasks where the token bill dwarfs the accuracy gain.
The failure mode to watch for is overthinking: on easy inputs a reasoning model can spend a huge budget second-guessing a correct first answer, adding cost and sometimes talking itself out of the right response. This is why per-request effort control and good routing — cheap model for easy traffic, reasoning model for the hard tail — matter as much as the reasoning itself.
Using it in practice
If you’re building on reasoning models, a few habits keep the trade favorable.
| Lever | What to do |
|---|---|
| Route by difficulty | Send easy traffic to a fast model; reserve reasoning (and its cost) for the hard tail. Most workloads are mostly easy. |
| Set the effort | Use the reasoning-effort / thinking-budget knob per request. Don’t default everything to high. |
| Budget the tail | Reasoning output is variable and can be large — model the token cost before you ship, especially inside agent loops. |
| Verify externally | For code/math, run the actual test or checker rather than trusting the model’s self-verdict. |
| Watch latency | Thinking tokens are real seconds. Stream a “thinking…” state, or precompute where you can. |
The mental model to keep: test-time compute is a dial, not a default. The teams that win with reasoning models aren’t the ones that turn it to maximum — they’re the ones that spend it precisely where checking beats guessing, and stay cheap everywhere else.
Worked example: voting vs a verifier, by the numbers
Take one question and two models. Model A gets it right on 60% of samples, model B on 30%. Each sample costs about 2,000 tokens. Assume the worst case for voting: every wrong sample gives the same wrong answer. Compare a majority vote with best-of-N under a verifier that always recognises a correct answer.
| Samples N | Tokens | A: vote | A: verifier | B: vote | B: verifier |
|---|---|---|---|---|---|
| 1 | 2,000 | 0.600 | 0.600 | 0.300 | 0.300 |
| 5 | 10,000 | 0.683 | 0.990 | 0.163 | 0.832 |
| 15 | 30,000 | 0.787 | > 0.999 | 0.050 | 0.995 |
| 45 | 90,000 | 0.914 | > 0.999 | 0.002 | > 0.999 |
Three lessons sit in that table. First, voting amplifies whatever the model already does most often. Above 50% it climbs; below 50% it confidently converges on the wrong answer, and B’s 0.300 falls to 0.002. Second, a verifier only needs the right answer to appear once: 1 − (1 − p)N, so 1 − 0.75 = 0.832 even for the weak model. That is the generate-and-verify gap in one line. Third, tokens grow linearly with N while accuracy flattens: going from 15 to 45 samples triples the bill for A’s vote and adds about 0.13. Real wrong answers are often scattered rather than identical, which makes voting better than this worst case, but the direction holds.
The cost of each way to spend it
Here Time is the wait the user sees and Space is the tokens billed per query. T is the tokens in one answer; the 50 tokens/s decode speed is illustrative.
| Case | Time | Space | Why |
|---|---|---|---|
| One answer, no thinking | One short generation | T | The baseline every other row is paying to beat. |
| Long chain-of-thought, budget B | Grows linearly with B: tokens come out one at a time. 2,000 thinking tokens at 50/s is 40 s; 8,000 is 160 s | B + T | The only row whose latency grows with its budget. Thinking cannot be parallelised. |
| Self-consistency, N samples | About one sample if the N run in parallel | N × (B + T) | Needs answers that can be compared exactly. Gains flatten once the majority is stable. |
| Best-of-N with a verifier | One sample, plus scoring | N × (B + T) + N verifier passes | The ceiling is the verifier. A test suite is nearly free; a reward model costs a forward pass each and can be fooled. |
| Search over steps (width w, depth d) | Grows with d: each level waits for the last | Roughly w × d step generations, plus a verifier call per step | Strongest when partial steps can be checked. It also costs the most and is the hardest to run. |
| A bigger model instead | Slower per token, on every request | Same token count, higher price per token | Pays on easy queries too, which is the trade test-time compute avoids by spending only where needed. |
The useful comparison is at equal spend. 8,000 tokens buys one 8,000-token chain (160 s) or four 2,000-token samples in parallel (about 40 s). Which wins depends on the task: long chains help when the problem needs many dependent steps, parallel samples help when a vote or checker can pick among short attempts.
Common mistakes
- Voting at temperature 0
Greedy decoding returns the same sample N times, so a vote of 8 is a vote of 1 at eight times the cost. Self-consistency needs diversity: sample at a non-zero temperature, then vote.
- Voting on free text
A majority vote needs answers you can compare exactly: a number, a letter, a test outcome. Two paragraphs never match word for word, so the votes split. Extract and normalise the final answer (
0.5,1/2and50%are one vote), or use a verifier instead. - Optimising against a weak verifier
Best-of-N picks whatever the scorer likes most. With a learned reward model and a large N, that is increasingly the answer that exploits the scorer’s blind spots: the convincing wrong one. Raise N only as far as the verifier stays trustworthy, and prefer checks that run, like tests.
- Forgetting the hidden tokens
On the major APIs, reasoning tokens are billed as output tokens even when you never see them. Where the output cap includes them, a tight cap can be spent entirely on thinking and return a cut-off or empty answer. Size the cap for thinking plus answer, and log reasoning tokens separately.
- Comparing at unequal compute
A reasoning model at high effort beating a base model in one shot proves little: one used many times the tokens. Compare at equal spend, as in the 8,000-token example above, or plot accuracy against tokens for both.
Where this shows up in production and interviews
- Code generation with tests
The clearest generate-and-verify setting. Sample several patches, run the test suite, keep one that passes. DeepMind’s AlphaCode took this to the extreme, sampling very large numbers of programs and filtering them on the problem’s example tests.
- The metrics in papers
Results tables report
pass@k(any of k samples correct, which is the perfect-verifier column above) andmaj@k(the majority of k, the vote column). Knowing which one a number is tells you whether a verifier is assumed. - Routers in front of models
Products put a cheap classifier or a small model in front, send the easy majority to a fast model and only the hard tail to a reasoning model at higher effort. The router’s errors become the system’s cost and quality problem, so it is evaluated like any other model.
- The interview version
“How would you raise accuracy without retraining?” Name the four levers, say which need a verifier, and do the equal-spend arithmetic: tokens, latency, and whether answers can be checked. Then name the failure: voting below 50%, or a verifier that can be gamed.
Check yourself
Your model is right 30% of the time on a multiple-choice task, and its wrong answers usually agree. You can afford 15 samples. Vote or verify?
Verify, if you can. With wrong answers agreeing, a 15-way vote converges on the common wrong choice: about 0.05 accuracy in the worst case. A reliable checker needs the right answer only once among 15, about 0.995. With no checker, extra samples are not worth buying here.
Should marketing copy for product pages go to a reasoning model at high effort?
Usually not. There is no checkable answer to verify against and no long chain of dependent steps, so the extra thinking buys little and adds latency and hidden output tokens. A fast model, or a light effort setting, fits.
Self-consistency at N = 8 scores exactly the same as N = 1. What are the two likely causes?
The samples are identical (temperature 0, so there is nothing to vote on), or the answers are not normalised, so equivalent answers like 1/2 and 0.5 split the vote and the winner is effectively random.
You double the thinking budget from 4,000 to 8,000 tokens. At 50 tokens/s, what happens to latency, and what is the parallel alternative at the same spend?
Thinking time goes from 80 s to 160 s, because thinking tokens are sequential. The same 8,000 tokens spent as four parallel 2,000-token samples plus a vote or checker takes about 40 s. That is better if short attempts can solve the task; the long chain is better if the task needs many dependent steps.
Quick answers
What is test-time compute?
Test-time (or inference-time) compute is the computation a model spends answering a question, as opposed to the compute used to train it. Reasoning models deliberately spend more of it — long chains of thought, many samples, or search — to trade extra inference cost for higher accuracy on hard problems.
What is a reasoning model?
An LLM trained to produce an extended internal chain of thought before its final answer, and to use that budget well. Models like o1/o3 and DeepSeek R1 are trained largely with reinforcement learning on verifiable rewards to think longer on hard problems, which sharply improves math, coding, and logic.
Does thinking longer always help?
No. It mainly helps problems with a real reasoning or search component. It does little for tasks bottlenecked on missing knowledge (use retrieval instead) or simple tasks a base model already handles, where it just adds latency and cost — and can even cause “overthinking” on easy inputs.
Is more test-time compute better than a bigger model?
For many hard tasks, yes — a smaller model that thinks can match or beat a bigger one that answers instantly. But there’s a compute-optimal balance: the strongest systems tune model size against thinking budget rather than maxing either. The two are complementary scaling axes.
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Transformers HandbookThe architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →