LLM Sampling & Decoding

Every knob that shapes how a language model picks its next token — and sensible presets.

Parameters

temperature: Flatten/sharpen the distribution. 0 = greedy, 1 = as-is, >1 = wilder.
top-k: Keep only the k highest-probability tokens, then renormalize.
top-p (nucleus): Keep the smallest set whose probability sums to p (e.g. 0.9).
min-p: Drop tokens below a fraction of the top token’s probability.
frequency penalty: Lower the odds of tokens by how often they’ve appeared.
presence penalty: Lower the odds of any token that has appeared at all.
max tokens: Hard cap on the length of the generated output.
stop sequences: Strings that end generation when produced.

Deterministic: temperature 0 — tools, extraction, tests
Balanced: temp 0.7, top-p 0.9 — general chat
Creative: temp 1.0+, top-p 0.95 — brainstorming, prose
Structured/JSON: temp 0 + a schema or grammar — never rely on temperature alone for valid JSON

Greedy: Always take the argmax token. Fast and repeatable, but prone to bland, looping text.
Beam search: Keep the top-b partial sequences by total probability. Good for translation/short exact outputs; costly and dull for open-ended text.
Sampling: Draw from the (temperature/top-p/top-k-shaped) distribution. The default for natural, diverse generation.
Speculative decoding: A small draft model proposes tokens the big model verifies in parallel — faster with identical output.

temp 0 ≠ fully deterministic: Batching, hardware, and MoE routing can still cause tiny nondeterminism across runs.
Stacking penalties: Heavy frequency + presence penalties can push the model into degenerate or off-topic text.
top-k vs top-p: Prefer top-p (nucleus): it adapts the cutoff to the distribution instead of a fixed count.