LLM Sampling & Decoding
Every knob that shapes how a language model picks its next token — and sensible presets.
At every step a language model outputs a logit for each token in its vocabulary; softmax turns those into probabilities, and a decoding strategy picks one token, appends it, and repeats. Every knob on this sheet either reshapes that distribution or changes how the pick is made.
So the knobs trade diversity against reliability: sharpen the distribution for extraction, tool calls and code; flatten or widen it for brainstorming and prose. They change how the model chooses among what it already believes — never what it knows.
[2.0, 1.0, 0.1] at three temperatures, drawn to scale. The order never changes, only the gaps.Parameters
- temperature
- Flatten/sharpen the distribution. 0 = greedy, 1 = as-is, >1 = wilder.
- top-k
- Keep only the k highest-probability tokens, then renormalize.
- top-p (nucleus)
- Keep the smallest set whose probability sums to p (e.g. 0.9).
- min-p
- Drop tokens below a fraction of the top token’s probability.
- frequency penalty
- Lower the odds of tokens by how often they’ve appeared.
- presence penalty
- Lower the odds of any token that has appeared at all.
- max tokens
- Hard cap on the length of the generated output.
- stop sequences
- Strings that end generation when produced.
- repetition penalty
- Hugging Face / llama.cpp style: scales down already-seen tokens; 1.0 is off, ~1.1–1.3 typical
- logit bias
- add a fixed amount to chosen token ids: −100 effectively bans a token
- seed
- best-effort repeatability for sampled outputs on some APIs — not a guarantee
- logprobs / top_logprobs
- return each token’s log probability and the top alternatives — for confidence and debugging
- typical-p
- keep tokens whose surprise is close to the distribution’s entropy — “locally typical” sampling
How the knobs combine
- softmax(z / T)
- temperature divides the logits before softmax: T below 1 sharpens, above 1 flattens
- Order of filters
- varies: Hugging Face applies penalties, temperature, then top-k / top-p; llama.cpp lets you reorder
- Temperature then top-p
- a higher temperature flattens the distribution, so more tokens fit inside the nucleus
- Entropy
- how spread out a step’s distribution is — high-entropy steps are where samples diverge
- Sequence log-probability
- the sum of token log-probs: every token lowers it, so search favors short outputs
- length penalty (beam)
- divide the score by length^α so beam search stops preferring short sequences
Presets
- Deterministic
temperature 0— tools, extraction, tests- Balanced
temp 0.7, top-p 0.9— general chat- Creative
temp 1.0+, top-p 0.95— brainstorming, prose- Structured/JSON
temp 0+ a schema or grammar — never rely on temperature alone for valid JSON- Code generation
temp 0–0.2; raise it only when sampling several candidates to test
Decoding strategies
- Greedy
- Always take the argmax token. Fast and repeatable, but prone to bland, looping text.
- Beam search
- Keep the top-b partial sequences by total probability — good for short exact outputs; dull for open text.
- Sampling
- Draw from the (temperature/top-p/top-k-shaped) distribution. The default for natural, diverse generation.
- Speculative decoding
- A small draft model proposes tokens the big model verifies in parallel — faster, same output distribution.
- Contrastive search
- favors likely tokens that differ from earlier context — fewer loops than greedy
- Diverse beam search
- beam groups penalized for overlapping — several genuinely different candidates
- Self-consistency
- sample several reasoning paths at temperature above 0, then majority-vote the final answer
- Best-of-n / reranking
- generate n candidates and keep the one a scorer or reward model rates highest
Structured output & stopping
- JSON mode
- guarantees syntactically valid JSON — not that it matches your schema
- JSON Schema outputs
- the provider masks every token that would break your schema during decoding
- Grammars (GBNF, regex)
- llama.cpp grammars, Outlines and XGrammar constrain open models the same way
- Tool calling
- the model fills arguments for a declared schema — still validate them before acting
- EOS token
- generation also ends when the model emits its end-of-sequence token
- finish_reason / stop_reason
length(OpenAI) ormax_tokens(Anthropic) means the output was cut off
Provider ranges
- OpenAI
temperature0–2 (default 1),top_p0–1, penalties −2.0 to 2.0- Anthropic
temperature0–1 (default 1), plustop_pandtop_k- Reasoning models
- some ignore or reject sampling parameters — check the model’s docs before tuning
- Tune one knob
- change temperature or top-p, not both at once — their effects compound
- max tokens vs context
- it caps output only; the prompt plus the output must still fit the context window
Gotchas
- temp 0 ≠ fully deterministic
- Batching, hardware, and MoE routing can still cause tiny nondeterminism across runs.
- Stacking penalties
- Heavy frequency + presence penalties can push the model into degenerate or off-topic text.
- top-k vs top-p
- Prefer top-p (nucleus): it adapts the cutoff to the distribution instead of a fixed count.
- Low temperature ≠ factual
- it makes the model more confident in its top guess, not more correct
- Repetition loops
- greedy and very low temperature can loop on long outputs — nudge temperature up or add a penalty
- Penalties hit code and JSON
- brackets and keywords must repeat; keep frequency and presence penalties near 0 there
Worked example: one sampling step with every knob on
The text so far is “The cat sat on the”. Five candidate tokens; settings frequency_penalty 0.5, presence_penalty 0.5, temperature 0.7, top_p 0.9. The token “cat” has already appeared once.
token logit after penalties p at T = 0.7 cumulative
mat 3.2 3.2 0.592 0.592
floor 2.6 2.6 0.251 0.844
sofa 2.1 2.1 0.123 0.967 reaches 0.9: cut here
cat 1.9 0.9 0.022
roof 0.4 0.4 0.011
after top_p, renormalized: mat 0.613 floor 0.260 sofa 0.127- Penalties. “cat” loses 1 × 0.5 for frequency plus 0.5 for presence, so 1.9 becomes 0.9.
- Temperature. Every logit is divided by 0.7, which widens the gaps: at T = 1 “mat” would get 0.490, at 0.7 it gets 0.592.
- top-p. Sorted by probability, the running sum first reaches 0.9 at “sofa”, so three tokens survive. “cat” and “roof” can no longer be drawn at all.
- Pick. A draw returns “mat” about 61% of the time, “floor” 26% and “sofa” 13%. Greedy decoding would return “mat” every time.
Change one setting and rerun the table to see its effect: at temperature 1.0 the cumulative sum after three tokens is 0.921, so the same three survive but “mat” drops to 53%.
Recipes
Temperature on three logits, worked through
logits z = [2.0, 1.0, 0.1]
T = 1.0: softmax(z) = [0.659, 0.242, 0.099]
T = 0.5: softmax(z / 0.5) = [0.864, 0.117, 0.019] sharper: the top token dominates
T = 2.0: softmax(z / 2.0) = [0.502, 0.304, 0.194] flatter: more variety
T -> 0: always the top token (greedy)Temperature never changes the ranking of tokens, only how far apart their probabilities sit — which is why it cannot make the model know something new.
top-p and top-k cutoffs, worked through
probabilities, sorted: A 0.50 B 0.25 C 0.15 D 0.06 E 0.04
cumulative: 0.50 0.75 0.90 0.96 1.00
top_p = 0.9 -> keep A, B, C (the cumulative sum reaches 0.90), renormalize:
A 0.556 B 0.278 C 0.167
top_k = 2 -> keep A, B, renormalize:
A 0.667 B 0.333top-p adapts: on a confident step it may keep a single token, on an uncertain one dozens. top-k always keeps exactly k, too many when the model is sure and too few when it is not.
min-p, worked through
min_p = 0.1 and the top token has 0.50 -> threshold = 0.1 * 0.50 = 0.05
keep A 0.50, B 0.25, C 0.15, D 0.06; drop E 0.04
confident step (top token 0.95): threshold 0.095 -> nearly everything else dropped
uncertain step (top token 0.20): threshold 0.020 -> many options surviveThe cutoff scales with the model’s confidence, which is why min-p tolerates higher temperatures without the tail of junk tokens creeping in.
Frequency and presence penalties, worked through
new_logit[j] = logit[j] - count[j] * frequency_penalty
- (1 if count[j] > 0 else 0) * presence_penalty
frequency_penalty = 0.5, presence_penalty = 0.5, every logit starts at 2.0
"the" seen 3 times: 2.0 - 3 * 0.5 - 0.5 = 0.0
"cat" seen once: 2.0 - 1 * 0.5 - 0.5 = 1.0
"dog" never seen: 2.0 (unchanged)This is the formula OpenAI documents. The frequency term grows with every repeat while the presence term is a one-off nudge toward new tokens; negative values encourage repetition instead.
Self-consistency: sample several answers, then vote
from collections import Counter
answers = []
for _ in range(5):
answers.append(ask(question, temperature=0.8)) # diverse reasoning paths
final = Counter(answers).most_common(1)[0][0] # majority voteIt works when the final answer is short and comparable, such as a number or a label, and it costs five times the tokens. At temperature 0 every sample would be the same, so the vote would add nothing.
When to use it, and when not
Sampling settings are the last thing to tune, not the first. Fix the prompt, the examples and the output format; pick a preset; then change one knob only if outputs are measurably too repetitive or too erratic across a test set.
- Leave them alone on reasoning models that ignore or reject them, and when a schema or grammar already constrains the output.
- Turn temperature down when there is one right answer: extraction, classification, tool arguments, SQL.
- Turn it up, with top-p or min-p as a guard, when you want different candidates: titles, brainstorming, or the samples for self-consistency and best-of-n.
- Use beam search only on open models, for short outputs with a narrow right answer, such as translating a sentence. OpenAI’s and Anthropic’s APIs do not expose it.
If the problem is wrong facts, no knob helps. That needs retrieval or a stronger model.
Trade-offs and limits
| Strategy | Generations | Diversity | Repeatable | Weak spot |
|---|---|---|---|---|
| Greedy | 1 | none | mostly | loops, bland text |
| Sampling | 1 | tunable | no; seed is best effort | an occasional junk token |
| Beam, b beams | 1, about b× the compute | low | yes | short, generic outputs |
| Self-consistency, n | n | by design | no | n× the cost; needs comparable answers |
| Speculative | 1 | unchanged | as the target model | speed-up depends on how many drafts are accepted |
| Schema-constrained | 1 | limited by the schema | as the sampler | guarantees structure, not correctness |
One trade-off hides inside constrained decoding: the model writes fields in schema order. Put a reasoning field before answer, or the answer is committed before any reasoning exists.
Common mistakes
- Banning a word with one token id. Logit bias works on tokens, and “ cat”, “cat” and “Cat” are usually different ids. Look up every variant in the model’s tokenizer, or the word slips through.
- Reading log-probabilities as accuracy. A token at −0.01 (p ≈ 0.99) means the model is sure, not that it is right. Use logprobs to find shaky spans, and calibrate against labelled data before treating them as a confidence score.
- Parsing output that was cut off. JSON that hit the token limit fails to parse, and line-by-line formats silently lose their last records. Check
finish_reasonorstop_reasonbefore parsing. - A stop sequence that occurs naturally. Stopping on
"\n\n"ends a multi-paragraph answer after its first paragraph, and the stop string itself is not returned. Use a marker that cannot appear in content, such as</answer>. - Judging a setting from one sample. One output per temperature is noise. Compare settings over a test set, with several samples per input.
Check yourself
Logits [2.0, 1.0, 0.1], temperature 1, top_p 0.8. Which tokens survive, with what probabilities?
Softmax gives 0.659, 0.242 and 0.099. The first alone is below 0.8; adding the second reaches 0.901, so two survive, renormalized to 0.731 and 0.269.
top_k 1 with temperature 1.5. What does the output look like?
Greedy. Only the highest-scoring token passes the filter, and temperature never changes the ranking, so the high temperature has no effect.
min_p 0.2; sorted probabilities 0.40, 0.30, 0.15, 0.09, 0.06. How many tokens stay?
Four. The threshold is 0.2 × 0.40 = 0.08, so only the 0.06 token is dropped.
Frequency penalty 0.4, presence penalty 0.6. A token with logit 1.5 has appeared 4 times. What is its new logit?
1.5 − 4 × 0.4 − 0.6 = −0.7. Two more appearances push it to −1.5: the frequency term keeps growing, the presence term is paid once.
Questions about LLM Sampling & Decoding Cheat Sheet
What does temperature do in an LLM?
Temperature rescales the model’s output scores before they are turned into probabilities: the logits are divided by the temperature and then passed through softmax. Values below 1 widen the gap between likely and unlikely tokens, so output becomes more predictable, and 0 is treated as always picking the top token. Values above 1 shrink the gap, giving rarer tokens a real chance, which increases variety but also mistakes. Temperature never changes which token the model ranks highest. Use about 0 to 0.3 for extraction, classification and code, and 0.7 to 1.0 for open-ended writing.
What is the difference between top-k and top-p sampling?
Both remove unlikely tokens before sampling, but they choose the cutoff differently. top-k keeps a fixed number of the most probable tokens, such as 40, no matter how the probability is spread. top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, such as 0.9, so the set shrinks when the model is confident and grows when it is unsure. That adaptivity is why top-p is usually preferred. They can be combined, and min-p is a newer alternative that sets the cutoff relative to the top token.
Does temperature 0 make an LLM deterministic?
Mostly, but not guaranteed. At temperature 0 the decoder always picks the highest-probability token, so the same input should give the same output. In practice, hosted models can still vary slightly between runs because of batching with other requests, non-deterministic GPU arithmetic, and routing in mixture-of-experts models; once two tokens are nearly tied, a tiny numeric difference flips the choice and the rest of the text diverges. Some APIs offer a seed parameter for best-effort repeatability. If you need exact reproducibility, cache outputs rather than relying on regeneration.
What is the difference between frequency penalty and presence penalty?
Both lower the scores of tokens that have already appeared, to reduce repetition. The frequency penalty grows with the number of times a token has been used, so a word that appears ten times is pushed down much harder than one that appeared once. The presence penalty is a flat, one-time deduction for any token that has appeared at all, which nudges the model toward new words and topics. In the OpenAI API both range from minus 2 to 2 and default to 0. Keep them small, because large values distort grammar, code and structured output.