Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LLM Sampling & Decoding

Every knob that shapes how a language model picks its next token — and sensible presets.

At every step a language model outputs a logit for each token in its vocabulary; softmax turns those into probabilities, and a decoding strategy picks one token, appends it, and repeats. Every knob on this sheet either reshapes that distribution or changes how the pick is made.

So the knobs trade diversity against reliability: sharpen the distribution for extraction, tool calls and code; flatten or widen it for brainstorming and prose. They change how the model chooses among what it already believes — never what it knows.

logits zpenalties+ biasz / Tsoftmaxtop-k/pmin-psampleor argmaxappend the token, repeat0.860.660.50T = 0.5T = 1T = 2
One decoding step. Penalties and bias edit the scores, temperature divides them, softmax turns them into probabilities, filters cut the tail, and one token is picked. Bars: the logits [2.0, 1.0, 0.1] at three temperatures, drawn to scale. The order never changes, only the gaps.

Parameters

temperature
Flatten/sharpen the distribution. 0 = greedy, 1 = as-is, >1 = wilder.
top-k
Keep only the k highest-probability tokens, then renormalize.
top-p (nucleus)
Keep the smallest set whose probability sums to p (e.g. 0.9).
min-p
Drop tokens below a fraction of the top token’s probability.
frequency penalty
Lower the odds of tokens by how often they’ve appeared.
presence penalty
Lower the odds of any token that has appeared at all.
max tokens
Hard cap on the length of the generated output.
stop sequences
Strings that end generation when produced.
repetition penalty
Hugging Face / llama.cpp style: scales down already-seen tokens; 1.0 is off, ~1.1–1.3 typical
logit bias
add a fixed amount to chosen token ids: −100 effectively bans a token
seed
best-effort repeatability for sampled outputs on some APIs — not a guarantee
logprobs / top_logprobs
return each token’s log probability and the top alternatives — for confidence and debugging
typical-p
keep tokens whose surprise is close to the distribution’s entropy — “locally typical” sampling

How the knobs combine

softmax(z / T)
temperature divides the logits before softmax: T below 1 sharpens, above 1 flattens
Order of filters
varies: Hugging Face applies penalties, temperature, then top-k / top-p; llama.cpp lets you reorder
Temperature then top-p
a higher temperature flattens the distribution, so more tokens fit inside the nucleus
Entropy
how spread out a step’s distribution is — high-entropy steps are where samples diverge
Sequence log-probability
the sum of token log-probs: every token lowers it, so search favors short outputs
length penalty (beam)
divide the score by length^α so beam search stops preferring short sequences

Presets

Deterministic
temperature 0 — tools, extraction, tests
Balanced
temp 0.7, top-p 0.9 — general chat
Creative
temp 1.0+, top-p 0.95 — brainstorming, prose
Structured/JSON
temp 0 + a schema or grammar — never rely on temperature alone for valid JSON
Code generation
temp 0–0.2; raise it only when sampling several candidates to test

Decoding strategies

Greedy
Always take the argmax token. Fast and repeatable, but prone to bland, looping text.
Beam search
Keep the top-b partial sequences by total probability — good for short exact outputs; dull for open text.
Sampling
Draw from the (temperature/top-p/top-k-shaped) distribution. The default for natural, diverse generation.
Speculative decoding
A small draft model proposes tokens the big model verifies in parallel — faster, same output distribution.
Contrastive search
favors likely tokens that differ from earlier context — fewer loops than greedy
Diverse beam search
beam groups penalized for overlapping — several genuinely different candidates
Self-consistency
sample several reasoning paths at temperature above 0, then majority-vote the final answer
Best-of-n / reranking
generate n candidates and keep the one a scorer or reward model rates highest

Structured output & stopping

JSON mode
guarantees syntactically valid JSON — not that it matches your schema
JSON Schema outputs
the provider masks every token that would break your schema during decoding
Grammars (GBNF, regex)
llama.cpp grammars, Outlines and XGrammar constrain open models the same way
Tool calling
the model fills arguments for a declared schema — still validate them before acting
EOS token
generation also ends when the model emits its end-of-sequence token
finish_reason / stop_reason
length (OpenAI) or max_tokens (Anthropic) means the output was cut off

Provider ranges

OpenAI
temperature 0–2 (default 1), top_p 0–1, penalties −2.0 to 2.0
Anthropic
temperature 0–1 (default 1), plus top_p and top_k
Reasoning models
some ignore or reject sampling parameters — check the model’s docs before tuning
Tune one knob
change temperature or top-p, not both at once — their effects compound
max tokens vs context
it caps output only; the prompt plus the output must still fit the context window

Gotchas

temp 0 ≠ fully deterministic
Batching, hardware, and MoE routing can still cause tiny nondeterminism across runs.
Stacking penalties
Heavy frequency + presence penalties can push the model into degenerate or off-topic text.
top-k vs top-p
Prefer top-p (nucleus): it adapts the cutoff to the distribution instead of a fixed count.
Low temperature ≠ factual
it makes the model more confident in its top guess, not more correct
Repetition loops
greedy and very low temperature can loop on long outputs — nudge temperature up or add a penalty
Penalties hit code and JSON
brackets and keywords must repeat; keep frequency and presence penalties near 0 there

Worked example: one sampling step with every knob on

The text so far is “The cat sat on the”. Five candidate tokens; settings frequency_penalty 0.5, presence_penalty 0.5, temperature 0.7, top_p 0.9. The token “cat” has already appeared once.

token   logit   after penalties   p at T = 0.7   cumulative
mat     3.2     3.2               0.592          0.592
floor   2.6     2.6               0.251          0.844
sofa    2.1     2.1               0.123          0.967   reaches 0.9: cut here
cat     1.9     0.9               0.022
roof    0.4     0.4               0.011

after top_p, renormalized:   mat 0.613   floor 0.260   sofa 0.127
  1. Penalties. “cat” loses 1 × 0.5 for frequency plus 0.5 for presence, so 1.9 becomes 0.9.
  2. Temperature. Every logit is divided by 0.7, which widens the gaps: at T = 1 “mat” would get 0.490, at 0.7 it gets 0.592.
  3. top-p. Sorted by probability, the running sum first reaches 0.9 at “sofa”, so three tokens survive. “cat” and “roof” can no longer be drawn at all.
  4. Pick. A draw returns “mat” about 61% of the time, “floor” 26% and “sofa” 13%. Greedy decoding would return “mat” every time.

Change one setting and rerun the table to see its effect: at temperature 1.0 the cumulative sum after three tokens is 0.921, so the same three survive but “mat” drops to 53%.

Recipes

Temperature on three logits, worked through

logits z = [2.0, 1.0, 0.1]

T = 1.0:  softmax(z)        = [0.659, 0.242, 0.099]
T = 0.5:  softmax(z / 0.5)  = [0.864, 0.117, 0.019]   sharper: the top token dominates
T = 2.0:  softmax(z / 2.0)  = [0.502, 0.304, 0.194]   flatter: more variety
T -> 0:   always the top token (greedy)

Temperature never changes the ranking of tokens, only how far apart their probabilities sit — which is why it cannot make the model know something new.

top-p and top-k cutoffs, worked through

probabilities, sorted:  A 0.50   B 0.25   C 0.15   D 0.06   E 0.04
cumulative:             0.50     0.75     0.90     0.96     1.00

top_p = 0.9 -> keep A, B, C (the cumulative sum reaches 0.90), renormalize:
               A 0.556   B 0.278   C 0.167
top_k = 2   -> keep A, B, renormalize:
               A 0.667   B 0.333

top-p adapts: on a confident step it may keep a single token, on an uncertain one dozens. top-k always keeps exactly k, too many when the model is sure and too few when it is not.

min-p, worked through

min_p = 0.1 and the top token has 0.50  ->  threshold = 0.1 * 0.50 = 0.05
keep A 0.50, B 0.25, C 0.15, D 0.06;   drop E 0.04

confident step (top token 0.95):  threshold 0.095 -> nearly everything else dropped
uncertain step (top token 0.20):  threshold 0.020 -> many options survive

The cutoff scales with the model’s confidence, which is why min-p tolerates higher temperatures without the tail of junk tokens creeping in.

Frequency and presence penalties, worked through

new_logit[j] = logit[j] - count[j] * frequency_penalty
                         - (1 if count[j] > 0 else 0) * presence_penalty

frequency_penalty = 0.5, presence_penalty = 0.5, every logit starts at 2.0
"the" seen 3 times:  2.0 - 3 * 0.5 - 0.5 = 0.0
"cat" seen once:     2.0 - 1 * 0.5 - 0.5 = 1.0
"dog" never seen:    2.0  (unchanged)

This is the formula OpenAI documents. The frequency term grows with every repeat while the presence term is a one-off nudge toward new tokens; negative values encourage repetition instead.

Self-consistency: sample several answers, then vote

from collections import Counter

answers = []
for _ in range(5):
    answers.append(ask(question, temperature=0.8))   # diverse reasoning paths
final = Counter(answers).most_common(1)[0][0]        # majority vote

It works when the final answer is short and comparable, such as a number or a label, and it costs five times the tokens. At temperature 0 every sample would be the same, so the vote would add nothing.

When to use it, and when not

Sampling settings are the last thing to tune, not the first. Fix the prompt, the examples and the output format; pick a preset; then change one knob only if outputs are measurably too repetitive or too erratic across a test set.

  • Leave them alone on reasoning models that ignore or reject them, and when a schema or grammar already constrains the output.
  • Turn temperature down when there is one right answer: extraction, classification, tool arguments, SQL.
  • Turn it up, with top-p or min-p as a guard, when you want different candidates: titles, brainstorming, or the samples for self-consistency and best-of-n.
  • Use beam search only on open models, for short outputs with a narrow right answer, such as translating a sentence. OpenAI’s and Anthropic’s APIs do not expose it.

If the problem is wrong facts, no knob helps. That needs retrieval or a stronger model.

Trade-offs and limits

StrategyGenerationsDiversityRepeatableWeak spot
Greedy1nonemostlyloops, bland text
Sampling1tunableno; seed is best effortan occasional junk token
Beam, b beams1, about b× the computelowyesshort, generic outputs
Self-consistency, nnby designnon× the cost; needs comparable answers
Speculative1unchangedas the target modelspeed-up depends on how many drafts are accepted
Schema-constrained1limited by the schemaas the samplerguarantees structure, not correctness

One trade-off hides inside constrained decoding: the model writes fields in schema order. Put a reasoning field before answer, or the answer is committed before any reasoning exists.

Common mistakes

  • Banning a word with one token id. Logit bias works on tokens, and “ cat”, “cat” and “Cat” are usually different ids. Look up every variant in the model’s tokenizer, or the word slips through.
  • Reading log-probabilities as accuracy. A token at −0.01 (p ≈ 0.99) means the model is sure, not that it is right. Use logprobs to find shaky spans, and calibrate against labelled data before treating them as a confidence score.
  • Parsing output that was cut off. JSON that hit the token limit fails to parse, and line-by-line formats silently lose their last records. Check finish_reason or stop_reason before parsing.
  • A stop sequence that occurs naturally. Stopping on "\n\n" ends a multi-paragraph answer after its first paragraph, and the stop string itself is not returned. Use a marker that cannot appear in content, such as </answer>.
  • Judging a setting from one sample. One output per temperature is noise. Compare settings over a test set, with several samples per input.

Check yourself

Logits [2.0, 1.0, 0.1], temperature 1, top_p 0.8. Which tokens survive, with what probabilities?

Softmax gives 0.659, 0.242 and 0.099. The first alone is below 0.8; adding the second reaches 0.901, so two survive, renormalized to 0.731 and 0.269.

top_k 1 with temperature 1.5. What does the output look like?

Greedy. Only the highest-scoring token passes the filter, and temperature never changes the ranking, so the high temperature has no effect.

min_p 0.2; sorted probabilities 0.40, 0.30, 0.15, 0.09, 0.06. How many tokens stay?

Four. The threshold is 0.2 × 0.40 = 0.08, so only the 0.06 token is dropped.

Frequency penalty 0.4, presence penalty 0.6. A token with logit 1.5 has appeared 4 times. What is its new logit?

1.5 − 4 × 0.4 − 0.6 = −0.7. Two more appearances push it to −1.5: the frequency term keeps growing, the presence term is paid once.

Questions about LLM Sampling & Decoding Cheat Sheet

What does temperature do in an LLM?

Temperature rescales the model’s output scores before they are turned into probabilities: the logits are divided by the temperature and then passed through softmax. Values below 1 widen the gap between likely and unlikely tokens, so output becomes more predictable, and 0 is treated as always picking the top token. Values above 1 shrink the gap, giving rarer tokens a real chance, which increases variety but also mistakes. Temperature never changes which token the model ranks highest. Use about 0 to 0.3 for extraction, classification and code, and 0.7 to 1.0 for open-ended writing.

What is the difference between top-k and top-p sampling?

Both remove unlikely tokens before sampling, but they choose the cutoff differently. top-k keeps a fixed number of the most probable tokens, such as 40, no matter how the probability is spread. top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, such as 0.9, so the set shrinks when the model is confident and grows when it is unsure. That adaptivity is why top-p is usually preferred. They can be combined, and min-p is a newer alternative that sets the cutoff relative to the top token.

Does temperature 0 make an LLM deterministic?

Mostly, but not guaranteed. At temperature 0 the decoder always picks the highest-probability token, so the same input should give the same output. In practice, hosted models can still vary slightly between runs because of batching with other requests, non-deterministic GPU arithmetic, and routing in mixture-of-experts models; once two tokens are nearly tied, a tiny numeric difference flips the choice and the rest of the text diverges. Some APIs offer a seed parameter for best-effort repeatability. If you need exact reproducibility, cache outputs rather than relying on regeneration.

What is the difference between frequency penalty and presence penalty?

Both lower the scores of tokens that have already appeared, to reduce repetition. The frequency penalty grows with the number of times a token has been used, so a word that appears ten times is pushed down much harder than one that appeared once. The presence penalty is a flat, one-time deduction for any token that has appeared at all, which nudges the model toward new words and topics. In the OpenAI API both range from minus 2 to 2 and default to 0. Keep them small, because large values distort grammar, code and structured output.