LLM Sampling Explained: Temperature, Top-k, Top-p (Nucleus), Min-p and Repetition Penalties for Production Generation (2026)
A language model never writes a sentence. It writes a probability distribution over roughly 100,000 tokens, and a few lines of decoding code pick one token from it, thousands of times per response. Those few lines decide whether your summariser is crisp or repetitive, whether your story generator is creative or incoherent, and whether a regression test at “temperature 0” passes twice in a row. Most teams still treat LLM temperature top-p settings as folklore, copied from a blog post and never tested. In 2026 that habit has a new cost: several flagship APIs now reject or ignore sampling parameters entirely, so code that sets them can fail with HTTP 400. This guide gives you the actual math, a worked six-token example, a NumPy implementation of each sampler, the failure modes, and the API limits as documented this month.
What this covers: softmax with temperature, greedy vs beam search, top-k, top-p (nucleus), min-p, locally typical sampling, Mirostat, OpenAI-style and CTRL-style repetition penalties, why “temperature 0” is still nondeterministic, current provider restrictions, and an illustrative settings table by task.
Context and Background
Decoding research went through a clear arc. Early neural machine translation used beam search, which keeps the B highest-scoring partial sequences and returns the best complete one. It works well when the target is narrow, as in translation. For open-ended text it fails in a now-famous way. Ari Holtzman and colleagues showed in “The Curious Case of Neural Text Degeneration” (ICLR 2020) that maximisation-based decoding produces bland, repetitive text, even from a strong model. Human text is not the most probable text; it regularly contains surprising tokens.
Pure random sampling has the opposite problem. A 50,000-token vocabulary has a long tail of individually unlikely tokens whose combined probability is not small. Sample from the full distribution for a few hundred steps and you will almost certainly draw from that tail, after which the text derails. The history of sampling since 2018 is a sequence of answers to one question: how much of the tail should be cut, and should the cut adapt to the model’s confidence?
Angela Fan, Mike Lewis and Yann Dauphin’s “Hierarchical Neural Story Generation” (ACL 2018) popularised top-k sampling, drawing “from the k=10 most likely candidates”. Holtzman et al. proposed nucleus (top-p) sampling, which adapts the cut to the shape of the distribution. Clara Meister and co-authors proposed locally typical sampling in TACL. Basu et al.’s Mirostat (ICLR 2021) turned the problem into feedback control of perplexity. Most recently, Minh Nhat Nguyen and co-authors published min-p sampling at ICLR 2025, a confidence-scaled threshold now available in Hugging Face Transformers and vLLM.
Meanwhile, hosted APIs moved the other way, and LLM temperature top-p controls became something providers restrict rather than expose. Reasoning models made sampling knobs less relevant to quality and more dangerous to expose. We cover which APIs restrict what in a later section. Because sampling operates on tokens, not words, it also helps to understand how BPE and SentencePiece tokenizers split text: a “repeated word” may be one token or five, which changes how penalties behave.
How LLM Sampling Works: From Logits to One Token
LLM sampling turns the model’s raw scores (logits) into one token per step. Logit processors first apply penalties and biases. Temperature then divides the logits, softmax converts them into probabilities, and truncation such as top-k, top-p or min-p removes unlikely tokens. The survivors are renormalised and a seeded random draw picks the token.

Figure 1: The per-token decoding loop used by most inference engines. Greedy decoding short-circuits the loop with an argmax.
Figure 1 shows the loop that runs once per generated token. The model produces a vector of logits, one per vocabulary entry. Logit processors adjust those numbers using what has already been generated. Temperature rescales them, softmax normalises them, and a truncation filter zeroes out part of the tail. The remaining probabilities are renormalised to sum to one, and a pseudo-random number generator draws the token. That token is appended to the context, repetition counts are updated, and the loop repeats. The order of these stages is not standardised, and it matters, as we show later. Every LLM temperature top-p setting you choose is an instruction to one of these stages, nothing more.
Temperature reshapes the distribution; it does not remove anything
Softmax with temperature is defined as p_i = exp(z_i / T) / Σ_j exp(z_j / T), where z_i is the logit for token i and T > 0. Dividing every logit by T scales the gaps between them. With T < 1 the gaps grow, so the top token’s share increases. With T > 1 the gaps shrink and the distribution flattens. As T approaches 0 the distribution collapses onto the argmax, which is why engines treat T = 0 as greedy decoding rather than dividing by zero. vLLM’s SamplingParams, for example, states that “Zero means greedy sampling” and resets top-p, top-k and min-p when temperature is effectively zero.
The key property for LLM temperature top-p tuning is that temperature never assigns zero probability. Every token keeps some mass, however tiny. At T = 1.5 on a 100,000-token vocabulary, the tail still exists; it is merely fatter. That is why temperature alone cannot prevent derailment, and why truncation samplers exist. In ratio terms, the odds between two tokens are exp((z_a − z_b)/T). A logit gap of 3.0 means odds of about 20:1 at T = 1, about 403:1 at T = 0.5, and only about 7.4:1 at T = 1.5.
Truncation decides which tokens are even eligible
Top-k keeps a fixed number of tokens. Top-p keeps the smallest set whose cumulative probability reaches p. Min-p keeps every token whose probability is at least p_base times the top token’s probability. Locally typical sampling keeps tokens whose surprisal is close to the distribution’s entropy. All four zero out the rest and renormalise.
The difference lies in what the cut adapts to. Top-k adapts to nothing; ten tokens is ten tokens whether the model is certain or confused. Top-p adapts to the shape of the distribution: when the model is confident, a handful of tokens reach 90% mass; when it is unsure, hundreds might be needed. Min-p adapts to the height of the peak, which is a direct signal of confidence. This is the core of the top-p vs top-k debate, and min-p is a third answer to it. In practice, LLM temperature top-p pairs are the most common configuration because the two stages complement each other: one reshapes, the other cuts.
Greedy vs beam search: why maximising is the wrong goal for open text
Greedy decoding takes the argmax at every step. Beam search approximates the most probable whole sequence. Both are deterministic in principle, and both are good when there is one right answer and the model is well calibrated, such as short extraction or classification-style outputs.
For long-form text, finding the mode of the distribution is actively harmful. Holtzman et al. found that the probability a model assigns to a repeated phrase grows with each repetition, so maximisation locks into loops. Felix Stahlberg and Bill Byrne showed in “On NMT Search Errors and Model Errors: Cat Got Your Tongue?” (EMNLP 2019) that exact search exposes a model flaw: “for more than 50% of the sentences, the model in fact assigns its global best score to the empty translation”. Beam search works partly because it fails to find the true mode. That is the strongest argument in the greedy vs beam search discussion: neither is a quality guarantee, and the better your search, the more model errors you expose. Beam search also multiplies decode compute by roughly the beam width and complicates streaming, which is why chat APIs do not offer it.
Walk-through: A Six-Token Worked Example and NumPy Implementation
Abstract definitions hide how differently these samplers behave. Take a toy step where a model has written “The robot picked up the” and produces logits for six candidate tokens. Real vocabularies have around 100,000 entries, but six is enough to see each mechanism. The logits are illustrative values chosen for this example, not output from a real model.
| Token | Logit | p at T=0.5 | p at T=0.7 | p at T=1.0 | p at T=1.5 | p at T=2.0 |
|---|---|---|---|---|---|---|
| box | 3.0 | 0.764 | 0.643 | 0.524 | 0.414 | 0.354 |
| cup | 2.2 | 0.154 | 0.205 | 0.235 | 0.243 | 0.237 |
| ball | 1.8 | 0.069 | 0.116 | 0.158 | 0.186 | 0.194 |
| wrench | 0.9 | 0.011 | 0.032 | 0.064 | 0.102 | 0.124 |
| banana | -0.5 | 0.001 | 0.004 | 0.016 | 0.040 | 0.062 |
| purple | -2.0 | 0.000 | 0.001 | 0.004 | 0.015 | 0.029 |
| Entropy (bits) | 1.06 | 1.44 | 1.78 | 2.09 | 2.25 |
All values were computed with the code below. Notice two things. First, temperature sampling in an LLM is a smooth dial: entropy rises from about 1.06 bits at T = 0.5 to 2.25 bits at T = 2.0. Second, “purple”, a token that is grammatical but semantically odd, climbs from effectively zero to about 3% at T = 2. Over a 500-token answer, a 3% per-step chance of a strange token compounds quickly.

Figure 2: Which tokens survive each truncation rule at T = 1 on the six-token example.
Figure 2 applies four truncation rules to the T = 1 column. Top-k with k = 3 keeps box, cup and ball. Top-p with p = 0.9 also keeps those three, because their cumulative mass is 0.524 + 0.235 + 0.158 = 0.917, the first prefix to reach 0.9. After renormalisation both give box 0.571, cup 0.257, ball 0.172. Min-p with p_base = 0.1 computes a threshold of 0.1 × 0.524 = 0.0524 and keeps four tokens, because wrench at 0.064 clears it. Locally typical sampling with τ = 0.5 does something surprising: it keeps cup and box but not in probability order.
Why typical sampling can skip the most likely token
Locally typical sampling ranks tokens by |−log p_i − H|, the distance between each token’s surprisal and the entropy H of the distribution. At T = 1 the entropy is about 1.232 nats. The surprisal of box is 0.647 nats, of cup 1.447 and of ball 1.847. So cup is the most “typical” token (distance 0.215), followed by box (0.585) and ball (0.615). The algorithm adds tokens in that order until their original probability mass reaches τ. With τ = 0.5, cup (0.235) plus box (0.524) reaches 0.759, so the set is {cup, box}.
With a very small τ such as 0.2, the set would be {cup} alone, and the most probable token could never be chosen. That is a deliberate design choice grounded in information theory: human text tends to carry information close to the expected rate. Meister et al. report τ = 0.2 working well for story generation and τ = 0.95 for abstractive summarisation. It is also a reason typical sampling is a poor fit for tasks with one correct answer.
Temperature changes what truncation sees
The interaction between the two stages is where most LLM temperature top-p advice goes wrong. Now raise the temperature to 2.0 and repeat. Cumulative mass in rank order becomes 0.354, 0.591, 0.786, 0.909. Top-p at 0.9 now keeps four tokens. Min-p at 0.1 has a threshold of 0.0354 and keeps five, admitting banana at 0.062. On this toy distribution, min-p is actually more permissive than top-p. That is honest and worth stating: with only six tokens there is no long tail, so the two methods look similar.
The difference appears with a realistic vocabulary. The table below uses a synthetic 50,000-token Zipf-shaped distribution, generated in NumPy purely to illustrate the mechanism. It is not a measurement of any real model.
| Synthetic case (illustrative) | T | Top token p | Tokens kept by top-p 0.9 | Tokens kept by min-p 0.1 |
|---|---|---|---|---|
| Uncertain, Zipf tail | 1.0 | 0.139 | 7,293 | 8 |
| Uncertain, Zipf tail | 1.5 | 0.016 | 34,353 | 23 |
| Uncertain, Zipf tail | 2.0 | 0.003 | 39,620 | 65 |
| Confident peak | 1.0 | 0.764 | 161 | 1 |
| Confident peak | 1.5 | 0.105 | 33,018 | 1 |
Top-p’s guarantee is about mass, not count. When the distribution is flat with a long tail, reaching 90% mass requires thousands of tokens, many of them nonsense. In the uncertain case at T = 1.5, about three-quarters of the nucleus mass sat beyond rank 100 in our synthetic run. Min-p’s guarantee is relative to the peak, so the eligible set stays small. This is the mechanism behind the Nguyen et al. claim that top-p “can still allow low-probability tokens into the sampling pool, leading to incoherent outputs” at high temperature, and why min-p is popular for creative generation at T ≥ 1.
The flip side is also visible. Min-p can be too strict when the model is confident but a second continuation is legitimate. In the confident case, min-p at 0.1 kept a single token at both temperatures, effectively becoming greedy. Whether that is good depends on your task.
NumPy implementation of each sampler
The following code implements every sampler discussed. It is a reference implementation for understanding and testing, not an optimised GPU kernel; production engines do this on batched tensors with fused kernels.
import numpy as np
def softmax(logits, temperature=1.0):
if temperature <= 0:
raise ValueError("use greedy() for temperature 0")
z = logits / temperature
z = z - z.max() # numerical stability
e = np.exp(z)
return e / e.sum()
def greedy(logits):
return int(np.argmax(logits))
def top_k_filter(probs, k):
if k <= 0 or k >= probs.size:
return probs
keep = np.argpartition(-probs, k - 1)[:k]
out = np.zeros_like(probs)
out[keep] = probs[keep]
return out / out.sum()
def top_p_filter(probs, p):
order = np.argsort(-probs)
cum = np.cumsum(probs[order])
cutoff = np.searchsorted(cum, p) + 1 # smallest prefix with mass >= p
out = np.zeros_like(probs)
out[order[:cutoff]] = probs[order[:cutoff]]
return out / out.sum()
def min_p_filter(probs, p_base):
threshold = p_base * probs.max()
out = np.where(probs >= threshold, probs, 0.0)
return out / out.sum()
def typical_filter(probs, tau):
logp = np.log(probs + 1e-12)
entropy = -(probs * logp).sum()
order = np.argsort(np.abs(-logp - entropy))
cum = np.cumsum(probs[order])
cutoff = np.searchsorted(cum, tau) + 1
out = np.zeros_like(probs)
out[order[:cutoff]] = probs[order[:cutoff]]
return out / out.sum()
def openai_penalties(logits, counts, frequency=0.0, presence=0.0):
return logits - counts * frequency - (counts > 0) * presence
def ctrl_repetition_penalty(logits, seen_mask, theta=1.0):
penalized = np.where(logits < 0, logits * theta, logits / theta)
return np.where(seen_mask, penalized, logits)
def sample(logits, rng, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0):
if temperature == 0:
return greedy(logits)
probs = softmax(logits, temperature)
probs = top_k_filter(probs, top_k)
if top_p < 1.0:
probs = top_p_filter(probs, top_p)
if min_p > 0.0:
probs = min_p_filter(probs, min_p)
return int(rng.choice(probs.size, p=probs))
if __name__ == "__main__":
vocab = ["box", "cup", "ball", "wrench", "banana", "purple"]
logits = np.array([3.0, 2.2, 1.8, 0.9, -0.5, -2.0])
p = softmax(logits)
print("top-k=3 ", np.round(top_k_filter(p, 3), 3))
print("top-p=0.9", np.round(top_p_filter(p, 0.9), 3))
print("min-p=0.1", np.round(min_p_filter(p, 0.1), 3))
print("typical.5", np.round(typical_filter(p, 0.5), 3))
rng = np.random.default_rng(seed=7)
print([vocab[sample(logits, rng, temperature=0.7, top_p=0.9)] for _ in range(10)])
Running it prints top-k and top-p distributions of [0.571, 0.257, 0.172, 0, 0, 0], a min-p distribution of [0.534, 0.24, 0.161, 0.065, 0, 0], and a typical distribution of [0.69, 0.31, 0, 0, 0, 0]. Two implementation details deserve attention. The z - z.max() subtraction prevents overflow in exp and does not change the result, because softmax is invariant to adding a constant to every logit. And top_p_filter uses “smallest prefix with mass ≥ p”, which always keeps at least one token; an off-by-one here produces an empty set and a crash when p is smaller than the top token’s probability.
Repetition, frequency and presence penalties
Penalties operate on logits before temperature, and there are two incompatible families. OpenAI’s advanced usage guide documents the additive form: mu[j] = mu[j] - c[j] * alpha_frequency - float(c[j] > 0) * alpha_presence, where c[j] counts how often token j has already been sampled. The presence penalty is a one-off subtraction for any token seen at least once. The frequency penalty grows linearly with the count. OpenAI suggests “around 0.1 to 1” to reduce repetition somewhat, up to 2 to strongly suppress it, warning that high values “can noticeably degrade the quality of samples”.
Apply it to our example. Suppose box has already appeared twice and cup once, with both coefficients at 0.5. Box’s logit falls from 3.0 to 3.0 − 1.0 − 0.5 = 1.5, and cup’s from 2.2 to 1.2. Ball, untouched at 1.8, becomes the most likely token at p ≈ 0.355, while box drops to about 0.263. That is the intended effect in prose. In code or JSON, it is a bug: penalising ", : or a variable name you must repeat corrupts output.
The second family is the multiplicative penalty from Salesforce’s CTRL paper (Keskar et al., 2019). It divides the logits of previously seen tokens by θ, and the authors report that “using a greedy sampling and θ ≈ 1.2 yields a good balance between truthful generation and lack of repetition”. Hugging Face Transformers implements it as torch.where(score < 0, score * penalty, score / penalty), so negative logits are multiplied to keep the penalty pointing downward. vLLM’s repetition_penalty applies to tokens in “the prompt and the generated text so far”, whereas its presence and frequency penalties consider only generated text. A July 2026 arXiv preprint by Peter Hollows argues that this sign-branched design depends on the arbitrary zero point of logits, reporting that it can flip many greedy tokens under a constant logit shift and cut valid JSON output sharply at a setting of 1.3. Treat those specific figures as a single, recent, not-yet-peer-reviewed result, but the underlying point follows directly from softmax’s shift invariance.
Nondeterminism, API Limits and Production Settings
Why “temperature 0” still gives different answers
Many teams assume greedy decoding is reproducible. Hosted APIs say otherwise. OpenAI’s documentation states that “Chat Completions are non-deterministic by default” and offers a seed parameter plus a system_fingerprint field, noting that determinism “may be impacted due to necessary changes OpenAI makes to model configurations on our end”. The more interesting question is why identical inputs produce different outputs even on one unchanged server.

Figure 3: How other users’ traffic changes your output through batch-size-dependent kernels.
The common explanation is “floating-point non-associativity plus GPU concurrency”. Horace He and colleagues at Thinking Machines Lab argued in “Defeating Nondeterminism in LLM Inference” (September 2025) that this is incomplete. A single forward pass is usually run-to-run deterministic. The real cause is that kernels are not batch-invariant: matrix multiplication, attention and RMSNorm choose different reduction strategies, such as split-K, depending on batch size. Your request’s batch size depends on everyone else’s traffic, so the floating-point summation order changes, logits change in their low bits, and an argmax on a near-tie flips. Figure 3 traces that chain.
Their experiment is concrete. They generated 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507 with the prompt “Tell me about Richard Feynman”, 1,000 tokens each. Standard vLLM produced 80 unique completions, diverging first at token 103. With batch-invariant kernels, all 1,000 were identical. The cost was real but bounded: on Qwen3-8B, a workload that took 26 seconds in default vLLM took 55 seconds unoptimised and 42 seconds with an improved attention kernel. In practice that means determinism is an engineering property of the serving stack, not a sampling parameter. A seed controls the random draw; it cannot fix logits that changed before the draw.
This matters beyond tests. Once a near-tie flips, everything after it is conditioned on a different prefix, so one bit-level difference can change a whole paragraph. If you run evaluations, run each prompt several times and report variance. Our guide to LLM evaluation harness architecture covers how to structure that.
Which APIs still accept sampling parameters in 2026
The most practical change for production teams is that hosted flagship models increasingly refuse sampling controls. As of this month’s documentation:
- Anthropic. The Claude thinking documentation states that on Claude Opus 4.7 and later models, including the Opus 5 family, Sonnet 5 family and Haiku 5.5, “non-default
temperature,top_p, ortop_kvalues return a 400 error on every request, regardless of whether thinking is used”. Thetop_pdefault there is 0.99, sotop_p: 1fails, as does sending temperature and top-p together. From API version2026-10-21, including any of the three at any value returns a 400 on every model. On older models and earlier API versions, temperature and top-k are incompatible with thinking, and top-p is allowed only between 0.95 and 1. - Google. The Gemini 3 developer guide says: “For all Gemini 3 models, we strongly recommend keeping the temperature parameter at its default value of 1.0.” It warns that lowering it “may lead to unexpected behavior, such as looping or degraded performance”, particularly in mathematical or reasoning tasks.
- OpenAI. Reasoning models have long rejected temperature with an “Unsupported parameter” error, as reported by developers on OpenAI’s community forum. The API reference says parameter support “can differ depending on the model used”, “particularly for newer reasoning models”, and points to the reasoning guide. Check the model page for the exact model you call; we could not find one consolidated official table.
- Self-hosted. vLLM, SGLang, llama.cpp and Hugging Face Transformers expose the full set, including min-p, typical sampling and repetition penalties.
Our reading of the design logic, which providers have not spelled out in full: reasoning models are post-trained to produce long chains of thought at a particular sampling setting. Lowering temperature can push them into the loops Holtzman described, and raising it can derail long reasoning. Providers now steer quality through effort levels and structured outputs instead. If you need guaranteed formats, use grammar-constrained decoding, which masks invalid tokens before sampling rather than hoping a low temperature avoids them.
Sampler order: a silent source of divergence
Engines differ in where temperature sits relative to truncation. The min-p paper computes its threshold “after applying temperature scaling”. In contrast, llama.cpp’s server documents a configurable samplers chain whose default, according to a third-party walkthrough of llama-server, ends with temperature after penalties, top-k, typical, top-p and min-p. If temperature runs last, truncation is computed on the T = 1 distribution and temperature only reshapes the survivors. A high temperature then cannot let tail tokens in, which makes “T = 1.5” much safer than it sounds.
The practical consequence: the same numbers produce different behaviour in different engines. Moving a prompt from a hosted API to llama.cpp or vLLM without re-tuning is a common source of “the open model is worse” complaints. Pin and record the full LLM temperature top-p configuration, plus every other sampler setting, including order, in your LLM observability traces.
Interaction with speculative decoding and structured output
Sampling also interacts with serving optimisations. Speculative decoding uses a cheap draft model to propose several tokens that the target model verifies in one pass. The original methods use a rejection-sampling acceptance rule designed to preserve the target model’s sampling distribution, so output quality should not change. Acceptance rates do change, however: at higher temperatures, draft and target distributions diverge more, fewer tokens are accepted, and the speedup shrinks. Our comparison of EAGLE-3, Medusa and lookahead decoding discusses acceptance rates in detail.
Constrained decoding interacts with truncation too. If a grammar mask removes all tokens that survived top-p, the engine must either fall back to the unmasked distribution or fail. Applying the grammar mask before truncation avoids this, but engines differ, so verify the order in yours. When you see JSON-mode failures under high temperature, check where your stack applies the mask.
Trade-offs, Gotchas, and What Goes Wrong
Tuning temperature and top-p together. Both control the effective breadth of the distribution, so changing both makes results hard to attribute. Pick one primary knob. A widely repeated LLM temperature top-p rule of thumb says the same: alter one, leave the other at its default. Anthropic’s newest models now enforce this by rejecting the pair outright.
Low temperature is not “more accurate”. It is more repetitive and more confident. On reasoning-tuned models, Google explicitly warns that temperature below 1.0 can cause looping. For a model trained to explore, the default is usually correct.
Top-k on its own is brittle. A fixed k of 40 is too wide when the model is certain and too narrow when ten continuations are equally valid. Use it as a safety cap alongside top-p or min-p, not as the main control.
Penalties corrupt structured output. Frequency and presence penalties count tokens, not meaning. Code repeats identifiers, JSON repeats quotes and braces, and tables repeat delimiters. A penalty that improves an essay will break a schema. Disable penalties for structured generation and use constrained decoding instead.
Repetition penalty includes the prompt in some engines. vLLM’s repetition_penalty considers prompt tokens; OpenAI’s penalties consider only sampled tokens. A RAG prompt full of product names can then suppress the very names the answer must contain.
High-temperature diversity hides quality collapse. Self-consistency voting and best-of-n reranking need diverse samples, so teams raise temperature. Past a point, more samples are diverse because they are wrong. Measure answer accuracy per sample, not just diversity.
Seeds are not a contract. A seed fixes the random draw on one exact stack. Model updates, hardware changes and batch composition all change logits. For audits, log the full output; do not plan to regenerate it.
Over-trusting toy benchmarks. Published sampler comparisons depend on model, temperature, task and evaluation method. Our synthetic table explains mechanisms; it does not predict your workload.

Figure 4: A decision flow for choosing sampling settings, starting with whether the provider allows them at all.
Practical Recommendations
Start by asking whether you should set LLM temperature top-p parameters at all. On current Anthropic flagship models and Gemini 3, the documented guidance is to leave them out or at default. Spend effort on prompts, reasoning effort and output schemas instead. On self-hosted open models, sampling remains a powerful lever, but it belongs in your evaluation loop, not in a config file nobody revisits.
The table below is an illustrative starting point, assembled from the sources cited above and common practice. It is not a benchmark result. Validate every row on your own held-out prompts.
| Task (illustrative) | Temperature | Truncation | Penalties | Notes |
|---|---|---|---|---|
| Extraction, classification, SQL | 0 (greedy) | none | off | Add constrained decoding for schemas |
| Code generation, single answer | 0 to 0.3 | top-p 0.95 | off | Use tests, not temperature, for correctness |
| Chat assistant on an open model | 0.6 to 0.8 | top-p 0.9 or min-p 0.05 | off or light | Tune one knob at a time |
| Creative writing, brainstorming | 1.0 to 1.5 | min-p 0.05 to 0.1 | presence 0.1 to 0.5 | Min-p paper suggests 0.05 to 0.1 |
| Self-consistency sampling | 0.7 to 1.0 | top-p 0.95 | off | Measure per-sample accuracy |
| Hosted reasoning model | provider default | provider default | not supported on some | Omit parameters entirely |
A short checklist:
- Check the model’s documentation before sending any sampling parameter; newer models may return a 400.
- Change one knob at a time: temperature or top-p, not both.
- Prefer min-p over top-p when running open models above T = 1.
- Disable repetition penalties for code, JSON and tables.
- Record LLM temperature top-p values, other truncation settings, sampler order, seed and system fingerprint with every trace.
- Run each evaluation prompt multiple times and report variance.
- Use batch-invariant kernels where bit-exact reproducibility is a hard requirement.
Frequently Asked Questions
What is the difference between top-p and top-k?
Top-k keeps a fixed number of the most likely tokens, such as 40, regardless of how the probability is spread. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose combined probability reaches p, such as 0.9. When the model is confident, top-p might keep only two tokens; when it is uncertain, hundreds. That adaptivity is why top-p usually beats top-k as the main control. Top-k remains useful as a hard cap that bounds the candidate set and limits worst-case tail sampling.
What LLM temperature top-p values should I use?
Use 0, or as low as the API allows, for tasks with one correct answer such as extraction or classification. Use around 0.6 to 0.8 for conversational output on open models, and 1.0 or higher with min-p for creative work. These are starting points, not rules. For hosted reasoning models, follow the provider: Google recommends keeping Gemini 3 at its default of 1.0, and Anthropic’s newest Claude models reject non-default temperature values. Always validate settings against your own evaluation set rather than copying them.
What is min-p sampling and why is it popular?
Min-p sampling keeps every token whose probability is at least a fraction of the top token’s probability. With p_base = 0.1 and a top token at 0.5, the threshold is 0.05. Because the threshold scales with model confidence, min-p stays strict when the model is sure and loosens when several continuations are plausible. Nguyen et al. presented it at ICLR 2025, reporting better quality and diversity at high temperatures than top-p, and recommending p_base between 0.05 and 0.1. Hugging Face Transformers and vLLM both support it.
Is greedy decoding better than beam search?
Neither is universally better. Greedy decoding picks the single most likely token at each step; beam search tracks several candidate sequences to find a higher-probability whole output. Beam search helps in narrow tasks such as translation, but for open-ended text both produce repetitive, bland output, as Holtzman et al. showed. Stahlberg and Byrne even found that exact search often returns an empty translation as the model’s top choice. Chat APIs therefore offer sampling, not beam search, and greedy remains useful for short deterministic outputs.
Why does temperature 0 give different outputs each time?
Temperature 0 makes the final choice deterministic, but the logits it chooses from can change between requests. Thinking Machines Lab showed that inference kernels such as matrix multiplication, attention and RMSNorm change their reduction order with batch size. Since batch size depends on other users’ traffic, logits shift slightly, and near-ties flip. In their test, 1,000 greedy completions of one prompt produced 80 distinct outputs until batch-invariant kernels were used. Hosted APIs add model and configuration updates on top, which a seed cannot control.
What is the difference between frequency penalty and presence penalty?
Both subtract from the logits of tokens that already appeared in the output. Presence penalty subtracts a fixed amount once a token has appeared at all, encouraging new topics. Frequency penalty subtracts an amount proportional to how many times the token has appeared, so heavy repetition is punished more. OpenAI documents both as additive changes to logits and suggests values around 0.1 to 1 for mild effects. A separate multiplicative repetition penalty, from the CTRL paper, divides positive logits of seen tokens and is used in many open-source engines.
Further Reading
- Constrained decoding with XGrammar, Outlines and llguidance — how grammar masks guarantee valid structure before a token is sampled.
- Speculative decoding compared: EAGLE-3 vs Medusa vs lookahead — why acceptance rates and speedups depend on your sampling temperature.
- LLM tokenization with BPE, SentencePiece and tiktoken — the token units that every sampler and penalty operates on.
- Ari Holtzman et al., “The Curious Case of Neural Text Degeneration”, ICLR 2020.
- Horace He and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, September 2025.
By Riju — about

Pingback: LLM Hallucination Detection in Production: 2026 Guide