Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS · 03  /  SAMPLING

The Dice Loader.

A language model doesn't pick the next word — it rolls for it, from a probability over every token. Three dials load those dice: temperature sharpens or flattens the odds, top-k keeps only the best few, top-p keeps just enough to cover the mass. Drive them and watch the bars move.

THE GIST · 20 SECONDS

The model emits a logit for every token; a softmax turns them into probabilities that sum to 1. Then it samples. Temperature divides the logits — low = sharp & safe, high = flat & wild. Top-k keeps the k highest and zeroes the rest. Top-p keeps the smallest set covering probability p. Whatever survives is renormalized back to 1 and rolled.

  • Softmaxlogits → probabilities
  • Temperaturesharpen ↔ flatten
  • Top-kkeep k best tokens
  • Top-pkeep p of the mass
Prompt: “The cat sat on the ___”

Drag the dials on the right. Coloured bars survive the filters and get renormalized; faded bars are cut. Hit Sample to roll.

UNDER THE HOOD

What you just played, written down

Every dial reshapes the same softmax distribution before the roll. Together they trade one thing against another: coherence versus creativity.

The three dials — what each does

  1. Softmax first. Logits become probabilities: p_i = e^(z_i / T) / Σ e^(z_j / T). That divisor T is temperature.
  2. Temperature = sharpness. T < 1 concentrates mass on the top tokens (safe, repetitive); T > 1 spreads it out (diverse, risky); T → 0 is greedy.
  3. Top-k = a hard cap. Keep the k most likely tokens, zero the rest, renormalize. Fixed count, ignores the distribution's shape.
  4. Top-p = an adaptive cap. Keep the smallest set whose probabilities sum to p, renormalize. Few survivors when confident, many when unsure.
WHY RENORMALIZE?

Once you zero out the cut tokens, the survivors no longer sum to 1 — you can't sample from that. Dividing each survivor by their total restores a valid distribution over just the tokens you kept.

The whole thing in code

import numpy as np

def sample(logits, temp=1.0, k=None, p=None):
    z = logits / temp                       # temperature
    probs = np.exp(z - z.max())
    probs /= probs.sum()                    # softmax

    order = probs.argsort()[::-1]           # high → low
    keep = set()
    cum = 0.0
    for rank, i in enumerate(order):
        if k is not None and rank >= k:     # top-k cap
            break
        keep.add(i); cum += probs[i]
        if p is not None and cum >= p:      # top-p cap
            break

    masked = np.array([probs[i] if i in keep else 0
                       for i in range(len(probs))])
    masked /= masked.sum()                  # renormalize
    return np.random.choice(len(masked), p=masked)
LOW TEMP / TIGHT psafefocused, repetitive
HIGH TEMP / LOOSE pcreativediverse, risky
TOP-P USUALLY WINS

Because top-p's survivor count adapts to each distribution, it's the common default. Top-k is simpler and caps the candidates hard. They stack — apply top-k, then top-p — on top of temperature.

⚠ Greedy isn't always best

Temperature 0 (always the top token) sounds "safe" but produces flat, looping text and can miss better continuations that start with a slightly lower-probability word. A little randomness usually reads better.

↔ Other decoders

Greedy takes the argmax; beam search keeps several partial sequences and maximizes total likelihood (great for translation, dull for chat); sampling with these dials is the norm for open-ended text.

★ Where it matters

The temperature and top_p parameters in most LLM APIs are exactly these dials. Tuning them is how you make a model deterministic for code or playful for brainstorming.

STEP BY STEP

Temperature, top-k and top-p, step by step

The widget's own ten logits for "The cat sat on the ___", pushed through each dial by hand, so the bars it draws are written out here.

The pipeline, in order

  1. Scale. Divide every logit by the temperature T.
  2. Softmax. Exponentiate and normalize; subtract the largest logit first so nothing overflows. Only differences matter: two tokens' odds are e(z_a − z_b)/T.
  3. Truncate. Top-k keeps the k most likely. Top-p walks down the sorted list and stops once the running total reaches p. Min-p keeps any token with at least m times the top token's probability.
  4. Renormalize and roll. Divide the survivors by their total, draw a uniform random number, and pick the token whose slice of the cumulative distribution it lands in.
# softmax with temperature
p_i = exp(z_i / T) / Σ_j exp(z_j / T)
# odds of two tokens
p_a / p_b = exp((z_a − z_b) / T)
# min-p keeps tokens with
z_max − z_i ≤ T · ln(1 / m)

Worked example: the cat on the mat

  1. Logits. mat 3.2, floor 2.1, sofa 1.8, roof 1.5, table 1.2, chair 0.9, bed 0.5, ground 0.2, couch −0.3, grass −0.8.
  2. T = 1, the starting view: mat 46.2%, floor 15.4%, sofa 11.4%, roof 8.4%, table 6.3%, down to grass under 1%. Entropy 2.47 bits. mat beats floor by e1.1 = 3.0 to 1.
  3. T = 0.5 doubles the gaps: mat 80.5%, floor 8.9%, odds 9.0 to 1, entropy 1.12 bits. T = 2 halves them: mat 25.1%, floor 14.5%, entropy 3.09 bits.
  4. Top-k = 3 at T = 1: mat, floor and sofa hold 73.0% of the mass and renormalize to 63.3%, 21.1% and 15.6%.
  5. Top-p = 0.9 at T = 1: the running total goes 46.2, 61.6, 73.0, 81.5, 87.7, 92.3%, so six tokens survive, mat at 50.1% down to chair at 5.0%. The same p keeps 3 tokens at T = 0.5 and 8 at T = 2.
  6. Min-p = 0.1 keeps tokens within ln 10 = 2.30 logits of the top at T = 1. chair, 2.3 below mat, survives by a hair at 0.1003 of mat's probability.

⚠ Order and interaction

Temperature applied before truncation changes which tokens survive, as the top-p row shows, and libraries do not all apply the dials in the same order. Tune temperature or top-p, not both at once; OpenAI's API reference gives the same advice.

↔ Temperature 0 is not a seed

Greedy decoding removes the dice, but a served model can still vary between identical requests, because batched GPU arithmetic can flip near-ties between logits. For repeatable evals, log the outputs rather than assuming determinism.

★ In production

Common starting points: T 0 to 0.2 for extraction, classification and code; T around 0.7 to 1.0 with top-p 0.9 to 0.95 for chat and writing. Frequency and presence penalties subtract from the logits of tokens already used, which curbs loops without flattening the whole distribution.

THE PICTURE

The picture: one roll, from logits to a token

“The cat sat on the ___” at T = 1, top-p = 0.9 logitsmat 3.2floor 2.1… grass −0.8 softmax z/TT = 1mat .462… grass .008 top-p 0.9sum 87.7 → 92.3%stop: keep 64 tokens cut renormalize÷ 0.923mat .501chair .050 mat .501 floor .167 sofa roof u = 0.62 → floor slices in order; table .068 and chair .050 fill the right end
Every dial acts before the roll. Temperature and softmax set the bar heights, top-p decides which bars exist, and renormalizing stretches the survivors back to a total of 1. The roll itself is one uniform random number dropped on the survivors laid end to end. Here it lands at 0.62, past mat’s slice (which ends at 0.501) and inside floor’s, so the model writes “floor” even though mat was the favourite. Roll again and mat comes up about half the time (50.1%); chair comes up about one roll in twenty.
TRADE-OFFS

Cost and trade-offs per decoding step

For a vocabulary of V tokens. All of these are cheap next to the forward pass that produced the logits. Beam search is the exception, because it multiplies the forward passes.

CaseTimeSpaceWhy
TemperatureO(V): one divide per logitO(1) extraChanges the shape, never which tokens are allowed.
SoftmaxO(V)O(V) probabilitiesSubtract the max first, as the page’s code does, or large logits overflow.
Greedy (argmax)O(V)O(1)No dice and no dials. The decoder adds no randomness, though a served model can still vary, as noted above.
Top-kO(V log k) with a size-k heapO(k)Never needs a full sort. A fixed count, blind to how peaked the distribution is.
Top-pO(V log V) if you sort everythingO(V)Adapts to the shape, but needs the tokens in order to walk the running total.
Min-pO(V): one pass against a thresholdO(1) extraAdaptive like top-p, with no sort: compare each token to a fraction of the top one.
Beam search, width BB forward passes per step, plus O(B·V)B copies of the KV cacheMaximises total likelihood. Strong for translation, bland for chat, and B times the serving cost.

So the choice between the sampling dials is about output quality, not speed. Only beam search changes the bill, because each beam is another sequence the model must run.

WATCH FOR

Common mistakes

  • Turning up temperature with top-k = 1

    Top-k = 1 keeps only mat, and renormalizing one token gives it 100%. Temperature then changes nothing, at 0.5 or 2. If a temperature change does nothing, look for a cap that leaves one survivor. Top-p can do it too: at T = 0.5 mat alone holds 80.5%, so top-p = 0.8 stops after one token.

  • Applying temperature to probabilities

    Temperature divides logits. Divide the probabilities by 0.5 instead and renormalize, and mat stays at 46.2%: the division cancels out. The right T = 0.5 gives mat 80.5%. On probabilities the equivalent is raising each to the power 1/T.

  • Expecting temperature to cut the junk

    The bottom four tokens hold 7.7% of the mass at T = 1 and 19.9% at T = 2. If every step looked like this one, a 20-token reply would pick at least one of them with odds of 79.6% at T = 1 and 98.8% at T = 2. Pair high temperature with top-p or min-p to trim the tail.

  • High temperature for structured output

    JSON, SQL and code break on a single wrong token, and every step is a roll. Use a low temperature there, and constrained decoding or a schema when the API offers one.

  • Judging a prompt from one sample

    At T = 0.8 one good answer may be luck. Compare prompts over several samples per input, or at T = 0 for the comparison, before deciding which is better.

QUICK CHECK

Did it stick?

See every answer, with the reasoning
  1. You lower the temperature toward 0. What happens to the distribution?

    Answer: It sharpens — probability concentrates on the top token(s) — Dividing logits by a small temperature exaggerates the gaps between them, so after softmax the top token dominates. At temperature ≈ 0 it becomes greedy — always the argmax.

  2. How does top-p (nucleus) differ from top-k?

    Answer: top-p keeps the smallest set covering p of the probability, so its count adapts to the distribution — top-k always keeps k tokens regardless of shape. top-p keeps however many tokens are needed to reach cumulative probability p — few when the model is confident, many when it is unsure.

  3. After top-k or top-p zeroes out some tokens, why renormalize the survivors?

    Answer: To make them sum back to 1 so they form a valid distribution to sample from — Zeroing tokens makes the remaining probabilities sum to less than 1, which is not a valid distribution. Dividing each survivor by their total restores a proper distribution over the kept tokens.

FAQ

Sampling, answered

How does a model pick the next token?

It outputs a logit per token; a softmax turns those into probabilities summing to 1; then it samples. Temperature, top-k, and top-p shape that distribution before the roll.

What does temperature do?

It divides the logits. Low (<1) sharpens the distribution toward the top tokens (safe, repetitive); high (>1) flattens it (diverse, risky); near 0 it's greedy.

What is top-k?

Keep only the k most likely tokens, zero the rest, and renormalize. A hard cap on how far into the tail the model can reach — but fixed regardless of the distribution's shape.

What is top-p (nucleus)?

Keep the smallest set of top tokens whose probabilities reach p, then renormalize. The survivor count adapts — few when the model is confident, many when it's unsure.

top-k or top-p?

Top-p adapts to each distribution and is usually the better default; top-k is simpler. They combine, with temperature, to trade coherence against creativity.

RESULT

—

Next → every temperature and top_p value in an LLM API is one of these dials. You now know exactly what they do to the dice.

Finished this one? 0 / 61 Labs done
NextChain-of-Thought vs Direct Next on the “Driving the model” ladder · rung 3 of 4

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs