Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Reasoning Models~15 min readIntermediate
The idea

Let the model think — the second scaling axis.

For years, better meant bigger: more parameters, more pretraining data. Reasoning models opened a second dial — spend more compute at inference, and a model that thinks can beat a much larger one that answers instantly. Here’s how test-time compute actually works, how o1- and R1-style models are trained to use it, and when it’s worth the latency.

01

What “test-time compute” means

Every model call spends compute. Train-time compute is what you burned once, up front, to create the weights. Test-time compute (or inference-time compute) is what you spend each time you answer a question. For a normal chat model, that per-answer cost is roughly fixed: it reads the prompt and emits a few hundred tokens.

A reasoning model breaks that assumption on purpose. Given a hard problem, it generates a long internal chain of thought — hundreds or thousands of “thinking” tokens — working the problem out step by step before committing to a final answer. It is deliberately spending more test-time compute to buy accuracy. The bet, borne out on math, coding, and logic benchmarks, is that for hard problems, a few extra seconds of thinking is worth more than a few extra billion parameters.

→ The one-line intuition

A base model answers from intuition, like a person blurting the first thing that comes to mind. A reasoning model gets to use scratch paper — and hard problems are exactly the ones where scratch paper matters.

This is why the same underlying model can be dramatically better or worse depending on how much you let it think. Thinking is no longer a fixed property of the model; it’s a budget you allocate per request.

02

Two scaling axes, not one

The last decade of progress came from scaling training. Test-time compute is a second, largely independent axis — and it has its own returns curve.

AxisWhat you scalePaidBest for
Train-timeParameters, data, pretraining FLOPsOnce, up frontBroad knowledge, fluency, everything a little better
Test-timeThinking tokens, samples, search per queryEvery requestHard, verifiable reasoning — math, code, logic, planning

The important research result is that these trade off against each other. On many hard tasks, a smaller model given a generous thinking budget matches or beats a much larger model answering in one shot — and the compute-optimal system is a balance of the two, not a maximum of either. If you have a fixed FLOPs budget to serve a workload, part of the art is deciding how much goes into a bigger model versus how much you leave for the model to think at inference.

There’s a reason this axis got attention exactly when it did: pretraining scaling is showing diminishing returns and is bounded by available data and cost. Test-time compute is a fresh lever with steep early returns on the problems that matter for agents and tools — a way to keep improving capability without training an ever-larger base model.

03

The four ways to spend it

“Think longer” is not one technique — it’s a family. They can be combined, and they trade cost for accuracy differently.

1 · Long chain-of-thought

  • One long generation: the model reasons step by step, backtracks, checks itself, then answers.
  • This is what o1/o3 and R1 do natively — the “thinking” is learned, not prompted.
  • Cost scales with how many thinking tokens it emits (the reasoning budget).

2 · Sampling + self-consistency

  • Generate N independent answers at temperature, then take the majority vote.
  • Works when the right answer is a stable attractor and wrong answers are scattered.
  • Cheap to implement; accuracy rises with N, then plateaus.

3 · Best-of-N with a verifier

  • Sample N candidates, then a verifier or reward model scores them and picks the best.
  • Far stronger than voting when you can check answers better than generate them.
  • Process reward models score each reasoning step, not just the final answer.

4 · Search

  • Explore a tree of partial reasoning steps, expanding promising branches (beam / MCTS-style).
  • Uses a verifier to guide where to spend the next unit of compute.
  • Most powerful and most expensive; shines on problems with checkable sub-steps.

The unifying principle is generate-and-verify: whenever checking a solution is easier than producing one, you can spend test-time compute producing many candidates and let verification find the good one. That gap between generation and verification difficulty is exactly what these methods exploit — and it’s why math and code, where answers are checkable, benefit most.

Accuracy vs test-time compute (schematic)
1 sampleBase accuracy — one shot, no thinking.
+ long CoTBig jump on reasoning tasks; the model works it out.
+ vote NAverages out lucky/unlucky samples; steady gains, then plateau.
+ verifierPicks the right one instead of the common one — higher ceiling.
+ searchBest ceiling on checkable problems; cost climbs fast.
Every method has a diminishing-returns knee. The art is stopping before the flat part.
04

How reasoning models learn to think

You can bolt sampling and verifiers onto any model. But models like o1/o3 and DeepSeek R1 go further: the long chain-of-thought is trained in, so a single generation reasons well on its own. The dominant recipe is reinforcement learning from verifiable rewards (RLVR).

The idea is elegant. Take problems with checkable answers — math with a known result, code with tests. Let the model generate a long reasoning trace, then reward it purely on whether the final answer is correct, with no human labeling the steps. Optimize that reward, and the model discovers, on its own, that thinking longer, checking its work, and backtracking lead to more correct answers. DeepSeek’s R1-Zero showed this emergent behavior striking: the reasoning length grew during training, and “aha”-style self-correction appeared without ever being explicitly taught.

→ Why verifiable rewards matter

A learned reward model can be gamed; a unit test cannot. Grounding the reward in something objectively checkable is what lets you crank up RL without the model learning to produce convincing-but-wrong reasoning. That’s the whole subject of the RLVR handbook.

The result is a model whose default mode is to spend test-time compute wisely — often exposed to you as a reasoning-effort setting (low / medium / high) that dials the thinking budget up or down per request.

05

When it helps — and when it just burns money

Test-time compute is not free capability; it’s a trade of latency and cost for accuracy, and the trade only pays on the right problems.

✓ Worth it

  • Competition math, hard coding, multi-step logic and planning.
  • Anything with a checkable answer, where generate-and-verify applies.
  • High-stakes single answers where being right dominates being fast.
  • Agentic tool use, where one bad step compounds.

✗ Wasteful

  • Simple lookups, formatting, chit-chat — a base model already nails these.
  • Knowledge the model lacks — thinking harder won’t invent facts; retrieval is the fix.
  • Latency-sensitive UX where seconds of “thinking” break the experience.
  • High-volume cheap tasks where the token bill dwarfs the accuracy gain.

The failure mode to watch for is overthinking: on easy inputs a reasoning model can spend a huge budget second-guessing a correct first answer, adding cost and sometimes talking itself out of the right response. This is why per-request effort control and good routing — cheap model for easy traffic, reasoning model for the hard tail — matter as much as the reasoning itself.

06

Using it in practice

If you’re building on reasoning models, a few habits keep the trade favorable.

LeverWhat to do
Route by difficultySend easy traffic to a fast model; reserve reasoning (and its cost) for the hard tail. Most workloads are mostly easy.
Set the effortUse the reasoning-effort / thinking-budget knob per request. Don’t default everything to high.
Budget the tailReasoning output is variable and can be large — model the token cost before you ship, especially inside agent loops.
Verify externallyFor code/math, run the actual test or checker rather than trusting the model’s self-verdict.
Watch latencyThinking tokens are real seconds. Stream a “thinking…” state, or precompute where you can.

The mental model to keep: test-time compute is a dial, not a default. The teams that win with reasoning models aren’t the ones that turn it to maximum — they’re the ones that spend it precisely where checking beats guessing, and stay cheap everywhere else.

Related concepts
Chain-of-thought Self-consistency Process reward models Best-of-N Compute-optimal scaling Speculative decoding
By the numbers

Worked example: voting vs a verifier, by the numbers

Take one question and two models. Model A gets it right on 60% of samples, model B on 30%. Each sample costs about 2,000 tokens. Assume the worst case for voting: every wrong sample gives the same wrong answer. Compare a majority vote with best-of-N under a verifier that always recognises a correct answer.

Samples NTokensA: voteA: verifierB: voteB: verifier
12,0000.6000.6000.3000.300
510,0000.6830.9900.1630.832
1530,0000.787> 0.9990.0500.995
4590,0000.914> 0.9990.002> 0.999

Three lessons sit in that table. First, voting amplifies whatever the model already does most often. Above 50% it climbs; below 50% it confidently converges on the wrong answer, and B’s 0.300 falls to 0.002. Second, a verifier only needs the right answer to appear once: 1 − (1 − p)N, so 1 − 0.75 = 0.832 even for the weak model. That is the generate-and-verify gap in one line. Third, tokens grow linearly with N while accuracy flattens: going from 15 to 45 samples triples the bill for A’s vote and adds about 0.13. Real wrong answers are often scattered rather than identical, which makes voting better than this worst case, but the direction holds.

Trade-offs

The cost of each way to spend it

Here Time is the wait the user sees and Space is the tokens billed per query. T is the tokens in one answer; the 50 tokens/s decode speed is illustrative.

CaseTimeSpaceWhy
One answer, no thinkingOne short generationTThe baseline every other row is paying to beat.
Long chain-of-thought, budget BGrows linearly with B: tokens come out one at a time. 2,000 thinking tokens at 50/s is 40 s; 8,000 is 160 sB + TThe only row whose latency grows with its budget. Thinking cannot be parallelised.
Self-consistency, N samplesAbout one sample if the N run in parallelN × (B + T)Needs answers that can be compared exactly. Gains flatten once the majority is stable.
Best-of-N with a verifierOne sample, plus scoringN × (B + T) + N verifier passesThe ceiling is the verifier. A test suite is nearly free; a reward model costs a forward pass each and can be fooled.
Search over steps (width w, depth d)Grows with d: each level waits for the lastRoughly w × d step generations, plus a verifier call per stepStrongest when partial steps can be checked. It also costs the most and is the hardest to run.
A bigger model insteadSlower per token, on every requestSame token count, higher price per tokenPays on easy queries too, which is the trade test-time compute avoids by spending only where needed.

The useful comparison is at equal spend. 8,000 tokens buys one 8,000-token chain (160 s) or four 2,000-token samples in parallel (about 40 s). Which wins depends on the task: long chains help when the problem needs many dependent steps, parallel samples help when a vote or checker can pick among short attempts.

Watch for

Common mistakes

  • Voting at temperature 0

    Greedy decoding returns the same sample N times, so a vote of 8 is a vote of 1 at eight times the cost. Self-consistency needs diversity: sample at a non-zero temperature, then vote.

  • Voting on free text

    A majority vote needs answers you can compare exactly: a number, a letter, a test outcome. Two paragraphs never match word for word, so the votes split. Extract and normalise the final answer (0.5, 1/2 and 50% are one vote), or use a verifier instead.

  • Optimising against a weak verifier

    Best-of-N picks whatever the scorer likes most. With a learned reward model and a large N, that is increasingly the answer that exploits the scorer’s blind spots: the convincing wrong one. Raise N only as far as the verifier stays trustworthy, and prefer checks that run, like tests.

  • Forgetting the hidden tokens

    On the major APIs, reasoning tokens are billed as output tokens even when you never see them. Where the output cap includes them, a tight cap can be spent entirely on thinking and return a cut-off or empty answer. Size the cap for thinking plus answer, and log reasoning tokens separately.

  • Comparing at unequal compute

    A reasoning model at high effort beating a base model in one shot proves little: one used many times the tokens. Compare at equal spend, as in the 8,000-token example above, or plot accuracy against tokens for both.

In the wild

Where this shows up in production and interviews

  • Code generation with tests

    The clearest generate-and-verify setting. Sample several patches, run the test suite, keep one that passes. DeepMind’s AlphaCode took this to the extreme, sampling very large numbers of programs and filtering them on the problem’s example tests.

  • The metrics in papers

    Results tables report pass@k (any of k samples correct, which is the perfect-verifier column above) and maj@k (the majority of k, the vote column). Knowing which one a number is tells you whether a verifier is assumed.

  • Routers in front of models

    Products put a cheap classifier or a small model in front, send the easy majority to a fast model and only the hard tail to a reasoning model at higher effort. The router’s errors become the system’s cost and quality problem, so it is evaluated like any other model.

  • The interview version

    “How would you raise accuracy without retraining?” Name the four levers, say which need a verifier, and do the equal-spend arithmetic: tokens, latency, and whether answers can be checked. Then name the failure: voting below 50%, or a verifier that can be gamed.

Self-check

Check yourself

Your model is right 30% of the time on a multiple-choice task, and its wrong answers usually agree. You can afford 15 samples. Vote or verify?

Verify, if you can. With wrong answers agreeing, a 15-way vote converges on the common wrong choice: about 0.05 accuracy in the worst case. A reliable checker needs the right answer only once among 15, about 0.995. With no checker, extra samples are not worth buying here.

Should marketing copy for product pages go to a reasoning model at high effort?

Usually not. There is no checkable answer to verify against and no long chain of dependent steps, so the extra thinking buys little and adds latency and hidden output tokens. A fast model, or a light effort setting, fits.

Self-consistency at N = 8 scores exactly the same as N = 1. What are the two likely causes?

The samples are identical (temperature 0, so there is nothing to vote on), or the answers are not normalised, so equivalent answers like 1/2 and 0.5 split the vote and the winner is effectively random.

You double the thinking budget from 4,000 to 8,000 tokens. At 50 tokens/s, what happens to latency, and what is the parallel alternative at the same spend?

Thinking time goes from 80 s to 160 s, because thinking tokens are sequential. The same 8,000 tokens spent as four parallel 2,000-token samples plus a vote or checker takes about 40 s. That is better if short attempts can solve the task; the long chain is better if the task needs many dependent steps.

Frequently asked

Quick answers

What is test-time compute?

Test-time (or inference-time) compute is the computation a model spends answering a question, as opposed to the compute used to train it. Reasoning models deliberately spend more of it — long chains of thought, many samples, or search — to trade extra inference cost for higher accuracy on hard problems.

What is a reasoning model?

An LLM trained to produce an extended internal chain of thought before its final answer, and to use that budget well. Models like o1/o3 and DeepSeek R1 are trained largely with reinforcement learning on verifiable rewards to think longer on hard problems, which sharply improves math, coding, and logic.

Does thinking longer always help?

No. It mainly helps problems with a real reasoning or search component. It does little for tasks bottlenecked on missing knowledge (use retrieval instead) or simple tasks a base model already handles, where it just adds latency and cost — and can even cause “overthinking” on easy inputs.

Is more test-time compute better than a bigger model?

For many hard tasks, yes — a smaller model that thinks can match or beat a bigger one that answers instantly. But there’s a compute-optimal balance: the strongest systems tune model size against thinking budget rather than maxing either. The two are complementary scaling axes.

Test-Time Compute & Reasoning Models · a Vibe Engines handbook · 2026
Finished this one? 0 / 209 Handbooks done
NextRL from Verifiable Rewards (RLVR) Next on the “Training and adaptation” ladder · rung 6 of 7

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks