Temperature & Sampling
A model never “knows” the next word — it scores every option, then picks one. Sampling is how it picks. Drag the knobs and watch the same model become precise, creative, or completely random.
🎲 The one-line intuition
Imagine a bag of weighted marbles. The word the model likes most gets the most marbles; less-likely words get fewer; then you reach in and grab one. Sampling settings just change how you fill the bag and how you reach in. Temperature controls how lopsided the marble counts are. Top-k and top-p throw out the unlikely marbles before you draw.
The model is choosing the next word for:
“The weather today is ___”
Reshapes how peaky the distribution is
Keep only the k most likely tokens
Keep the smallest set summing to p
Probability of each next word
8 of 8 tokens in playBlue bars are the tokens still eligible to be sampled. Greyed, struck-through tokens were trimmed by top-k / top-p. Percentages are after temperature, before renormalizing the survivors.
Determinism meter
VariedPress “Roll the dice” to draw a word weighted by the current distribution. Roll many times and watch the frequencies match the probabilities.
Step by step
How a model picks the next word
Logits
The model outputs one raw score — a “logit” — for every possible next word. Logits are just numbers; higher means the model likes it more. A logit of 12 doesn't mean 12% — it only means “more than 8”.
Softmax
Softmax turns those raw scores into real probabilities that add up to 1: raise e to each score, then divide by the total. Because it's exponential, small score gaps become big probability gaps.
Distribution
Now there's a ranked menu over the whole vocabulary — e.g. sunny 35%, cloudy 17%, rainy 12%, and a long tail of tiny probabilities.
Sample
Draw one word, weighted by those probabilities. The model doesn't “decide” — it rolls weighted dice. Same distribution, different rolls, different words. That's why one prompt can give many answers.
logits → ÷ temperature → softmax → top-k → top-p → renormalize → sample
The three knobs
🌡️ Temperature
Divides every logit before softmax. Small T stretches the gaps (peaky, confident); large T shrinks them (flat, random).
- T = 0 — greedy, always the top word
- T < 1 — sharper, safe (facts, code)
- T = 1 — the model's raw distribution
- T > 1 — flatter, creative → eventually gibberish
🎯 Top-k
Keep only the k highest-probability words, throw the rest away, renormalize, then sample. A hard cutoff by count.
Guarantees you never sample from the garbage tail — but k is fixed regardless of how confident the model is.
🪣 Top-p (nucleus)
Add words from the top until their cumulative probability reaches p, keep exactly that set, renormalize, sample.
Keeps a variable number of words: tiny when the model is sure, wide when it's torn. That's why it adapts better than top-k.
Which temperature when
Rule of thumb: one right answer → low temperature; many good answers → higher temperature. Set it per task, not once globally.
| Use case | Temperature | Why |
|---|---|---|
| Code generation | 0.0 – 0.2 | Syntax must be exact — take the safe token |
| Factual Q&A | 0.0 – 0.3 | Consistent, reproducible answers |
| Summarization | 0.3 – 0.5 | Faithful, with a little phrasing variety |
| Conversational chat | 0.7 – 1.0 | Natural and varied, still coherent |
| Creative writing | 0.8 – 1.2 | Expressive and imaginative |
| Brainstorming | 1.0 – 1.4 | Maximize diversity of ideas |
The intuitions people miss
Six things that surprise people
Temperature hits logits, not probabilities
It divides the raw scores before softmax. Reshaping already-computed probabilities would be a different, wrong operation.
Temperature 0 isn't always perfectly repeatable
In principle it's argmax. In practice, floating-point math on parallel hardware can flip near-ties, so you may see rare variation.
High temperature can produce gibberish
Flattening the distribution eventually gives nonsense words a real chance. “More creative” and “more broken” are the same knob past a point.
top-p = 1 turns nucleus filtering off
People set it thinking it's the “safe” value — but it actually disables the filter entirely and samples from the full distribution.
Greedy decoding can loop
Always taking the single most likely token often falls into “the best the best the best…” ruts. A little randomness breaks the loop.
Big top-k on a confident model does nothing
If the model is 99% sure, top-k=50 still admits 49 near-zero words you'll almost never draw. This is the gap top-p closes.
Related reading · LLM Fundamentals