TL;DR: The model emits a probability distribution over the next token; decoding is how you pick from it. Temperature rescales the distribution (low = sharper, high = flatter); top-k samples from the k most likely tokens; top-p (nucleus) samples from the smallest set whose cumulative probability exceeds p, so it adapts to the model's confidence. Use near-greedy for factual/structured tasks (extraction, code, classification) and higher temperature with top-p for creative generation.
How to approach it. Frame all three as ways to turn the next-token distribution into a concrete choice, define each precisely by what it does to that distribution, then land the task-dependent recommendation. "Which settings for what" is the applied signal the interviewer is after.
A strong answer. At each step the model produces logits, which softmax turns into a probability over the vocabulary. Decoding selects a token from that distribution:
- Greedy decoding always takes the argmax. Deterministic and fine when there is one right answer, but it produces bland, repetitive text on open-ended generation and can lock into loops. That failure has a name and a fix list of its own: repetition and neural text degeneration.
- Temperature divides logits by T before softmax. Low T (< 1) sharpens toward the top tokens (conservative); high T (> 1) flattens the distribution (diverse, riskier); T approaching 0 reduces to greedy. It is the global creativity dial.
- Top-k sampling restricts sampling to the k highest-probability tokens (renormalized), cutting the long nonsensical tail. Simple, but a fixed k is sometimes too tight (when many tokens are reasonable) and sometimes too loose (when one token should dominate).
- Top-p / nucleus sampling keeps the smallest set of top tokens whose cumulative probability exceeds p (say 0.9), then samples. It adapts to confidence: when the model is sure, the nucleus shrinks toward greedy; when unsure, it widens. That adaptivity is why top-p usually beats top-k.
These compose. Temperature reshapes the distribution; top-k/top-p truncate the candidate set you then sample from.
| Task type | Temperature | Truncation | Why |
|---|---|---|---|
| Extraction, classification, code | 0 to 0.3 | greedy or small top-p | reproducible, conservative, fewer avoidable errors |
| Summarization, chat | 0.5 to 0.7 | top-p ~0.9 | fluent but controlled |
| Brainstorming, creative writing | 0.8 to 1.0 | top-p ~0.9 to 0.95 | diversity, surprise |
The defensible point: top-p adapts the candidate set to confidence (its edge over fixed top-k), factual and creative tasks want opposite ends of the dial, and matching the setting to the task is the practitioner's judgment.
Key takeaways.
- Temperature rescales the whole distribution; top-k/top-p truncate the candidate set. They are orthogonal and combine.
- Top-p is confidence-adaptive (narrow when the model is sure, wide when unsure), which is why it is the usual default over fixed top-k.
- Factual and structured tasks want near-greedy; creative tasks want T around 0.7 to 1.0 with top-p.
- Reproducible output (evals, structured extraction) needs T = 0 plus a fixed seed where the stack allows.
What interviewers probe next.
- "Top-k vs top-p, why prefer top-p?" Top-p adapts the candidate set to the model's confidence; a fixed k is sometimes too tight, sometimes too loose.
- "How do you get deterministic output?" Temperature 0 / greedy and fix the seed where the stack supports it; needed for reproducible evals and structured extraction.
- "Why does greedy repeat itself?" It always takes the locally most probable token, which can trap it in high-probability loops; sampling or a repetition penalty breaks that.
- "Relationship to beam search?" Beam search maximizes sequence likelihood (good for translation/ASR); for open-ended generation it is bland, so sampling (top-p) wins.
Common mistakes.
- Confusing temperature (rescales the distribution) with top-k/top-p (truncate the candidate set); they compose rather than compete.
- Using high temperature for tasks that need correctness (code, extraction), causing avoidable errors.
- Treating top-k as strictly better than top-p; top-p's confidence-adaptivity is usually the win.
- Forgetting that reproducibility needs temperature 0 and a fixed seed, for example in evals.