Temperature and Sampling
The parameters that control how an LLM picks its next token: temperature scales the probability distribution, top_p and top_k trim it. Low values favour consistency, high values favour variety.
At every step of generation, an LLM produces a probability for every token in its vocabulary and then picks one. Sampling is that picking process, and temperature, top_p, and top_k are the dials that control it. They are set per request in the API call, and choosing them well is one of the cheapest quality improvements available: no prompt change, no model change, one parameter.
The dials
Temperature rescales the probability distribution before picking. Values below 1 sharpen it, so likely tokens become even more likely and the output gets more focused and repeatable. Values above 1 flatten it, so unlikely tokens get more chances and the output gets more varied. At temperature 0 the model picks the single most likely token at every step, called greedy decoding.
top_p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples within that set. With top_p 0.9, the model samples from the top 90% of probability mass and discards the long tail of unlikely tokens.
top_k keeps only the k most likely tokens and samples among them. top_k 40 means the 41st-most-likely token can never be picked.
Providers recommend adjusting temperature or top_p, not both at once, because they interact: both trim or spread the same distribution, and combined effects are hard to reason about.
Which value for which task
| Low (0 to 0.3) | Medium (0.5 to 0.8) | High (0.9+) | |
|---|---|---|---|
| Behaviour | Focused, repeatable | Balanced | Varied, surprising |
| Good for | Extraction, classification, code | General chat, drafting | Brainstorming, fiction |
| Risk | Repetitive phrasing | Middle ground | More hallucination-prone drift |
The determinism caveat
Temperature 0 makes output more repeatable, not guaranteed identical. Providers document that even greedy decoding can vary slightly between runs, because of floating-point non-determinism on parallel hardware and serving-stack differences. Treat temperature 0 as “as consistent as this model gets,” not as a bit-identical contract. If your test suite asserts exact string equality on model output, it will flake; evaluate meaning, structure, or score instead, as covered in From Deterministic Code to LLM Systems .
Where you set it
Both major APIs accept these as top-level request parameters, documented in the Anthropic Messages API reference and the OpenAI API reference . If you do not set them, the provider’s defaults apply, which are tuned for general chat rather than for your task.
Reasoning models are the main exception. They often restrict or ignore sampling parameters, because the provider tunes sampling for the hidden reasoning phase. For example, OpenAI’s GPT-6 models reject temperature, top_p, and top_logprobs whenever reasoning effort is set above none (OpenAI GPT-6 guide
). On those models, steer output through the prompt, the reasoning-effort setting, and structured output instead, and check each model’s documentation before relying on temperature.
Further reading
- Your First LLM API Call : see the parameters in a real request.
- From Deterministic Code to LLM Systems : why sampling changes how you test.
- Prompt Engineering : the other half of controlling output quality.
- Anthropic Messages API reference : official parameter documentation.
- OpenAI API reference : the same parameters on the OpenAI side.