Reasoning-Effort Control in LLM Serving: Thinking Budgets Explained

For two years the standard way to make a language model smarter on a hard request was to send that request to a bigger model. That reflex is now obsolete. Reasoning effort control turns a single model into a continuum of price/latency/quality operating points, and every major serving stack in 2026 exposes a knob for it — OpenAI’s reasoning.effort, Anthropic’s budget_tokens and its adaptive effort successor, Gemini’s thinking_level, and vLLM’s per-request thinking_token_budget. The consequence most teams have not absorbed is architectural: once effort is a request parameter, model selection stops being a design-time decision and becomes a runtime scheduling problem. You are no longer choosing a model. You are writing a policy.

What this covers: the actual parameter surfaces and their units, why variable-length thinking wrecks tail latency and KV-cache planning, how to build an effort router, what truncation does to a half-finished thought, the unit economics of reasoning tokens, and how to tell whether more thinking helps your task at all.

Context and Background

The first generation of reasoning models shipped as separate SKUs. You picked o1 or you picked GPT-4o; you picked a thinking model or a chat model, and that choice was baked into your service configuration. Cost control meant routing between endpoints, which is the pattern we covered in test-time compute scaling with OpenAI’s o3 family.

That separation is gone. The 2026 generation collapses both modes into one set of weights and exposes depth as a parameter. OpenAI’s Responses API documents reasoning.effort with model-dependent values that can include none, minimal, low, medium, high and xhigh, with gpt-5.5 defaulting to medium (OpenAI reasoning guide). Anthropic’s Claude models moved from a manual budget_tokens integer to adaptive thinking driven by an effort parameter — on Claude Opus 4.7, Opus 4.8, Sonnet 5, Fable 5 and Mythos 5, passing budget_tokens now returns a 400 error (Anthropic extended thinking docs). Google’s Interactions API uses thinking_level, with per-model supported sets and defaults that differ by model.

The open-weight side moved in the same direction but with different mechanics. Qwen3 exposed a hard switch, chat_template_kwargs: {"enable_thinking": false}, plus soft /think and /no_think prompt markers. Thinking Machines Lab’s Inkling, released 15 July 2026 under Apache 2.0, went further: a continuous effort dial the release post sweeps from 0.2 to 0.99, trained by varying the system message and the per-token cost across RL rollouts so the model learned to modulate its own trace length.

The strategic claim in the Inkling release is the one worth sitting with. On Terminal Bench 2.1, Inkling reaches Nemotron 3 Ultra’s score at roughly a third of the generated tokens. If that generalises even partially, the interesting axis of competition is no longer peak benchmark score. It is the shape of the score-versus-tokens curve — and curves are things schedulers can exploit.

Reasoning Effort Is a Scheduling Dimension, Not a Model Choice

Reasoning effort control is the ability to set, per request, how many tokens a model may spend on internal reasoning before it produces a visible answer. It is exposed as a categorical level (low/medium/high), an integer token budget, or a continuous scalar. Because the setting changes generation length rather than weights, it moves cost, latency and quality together along one axis on a single deployed model.

That last sentence is the whole argument. A knob that changes generation length is not an API convenience — it is an input to the scheduler. Batch composition, KV-cache residency, admission control and queueing all depend on how long a sequence will run, and effort is now the dominant predictor of that duration.

Figure 1: Effort is assigned from request features, dispatched to a pool sized for that budget class, and the realised reasoning-token counts feed back into the policy.

The diagram separates two things that usually get conflated. The policy decision (what effort does this request deserve) is a classification problem over request features. The placement decision (which pool serves it) is a capacity problem. Keeping them separate is what lets you downgrade effort under load without changing your routing topology, and lets you resize pools without retraining the classifier.

The parameter surface is not a standard, and the units differ

Treating these knobs as interchangeable is the first mistake. They are not even the same kind of quantity.

OpenAI’s reasoning.effort is a hint, not a cap. It biases how much the model thinks, and the docs note that models also reason adaptively within an effort level — spending fewer tokens on simple prompts even at high. The hard cap is a separate parameter, max_output_tokens, which bounds reasoning plus visible output together.

Anthropic’s budget_tokens, where still supported, is an integer with a documented minimum of 1,024 that must be less than max_tokens — except under interleaved thinking with tools, where it represents the total across all thinking blocks in an assistant turn and may exceed max_tokens. Critically, the docs describe the budget as a target rather than a strict limit, and note the model may not use the whole allocation, especially above 32k.

Gemini’s thinking_level is categorical, and the supported set is per-model: gemini-3-pro-preview accepts only low and high, while gemini-3.5-flash accepts minimal, low, medium and high and defaults to medium. Code that assumes a universal four-level ladder will throw on some models.

Inkling’s effort is a float. vLLM’s thinking_token_budget is a hard integer enforced by the engine. Five vendors, five semantics: hint, soft target, categorical, continuous, hard cap. A routing layer that abstracts over them needs a normalisation table and an honest acknowledgement that the mapping is lossy.

Variable-length thinking is a tail-latency problem, not an average-latency problem

The mean is a liar here. Effort control does not shift a latency distribution uniformly; it changes the distribution’s shape, and it does so in a way that punishes the naive SLO.

A non-thinking response has generation length roughly bounded by the answer’s natural length — usually a few hundred tokens with modest variance. A thinking response at high effort can emit tens of thousands of reasoning tokens before the first visible character, and the variance across semantically similar prompts is large because trace length depends on how many dead ends the model explores. Two prompts that look identical to your classifier can differ by an order of magnitude in reasoning tokens.

This breaks time-to-first-token as a metric. Under extended thinking, TTFT measures when reasoning starts, not when the user sees anything useful. The meaningful metric is time-to-first-visible-token, and it is the one your SLO should name. Anthropic’s display: "omitted" option exists precisely for this: it skips streaming thinking content so the final text begins streaming sooner. The docs are blunt that this reduces latency, not cost — you are still billed for the full thinking tokens.

There is a second-order effect that catches teams during incidents. Long reasoning traces occupy a decode slot for a long time, so a burst of high-effort requests does not just slow itself down; it starves the queue behind it. A p50 that looks fine can coexist with a p99 that has collapsed, because a small fraction of maximally-thinking requests is holding a disproportionate share of decode capacity. Effort mix, not request rate, is the load variable that matters.

Long traces are a KV-cache and batching problem

Continuous batching assumes you can pack many sequences into a batch and that per-sequence KV-cache footprint grows predictably. Reasoning tokens are ordinary generated tokens — they occupy KV-cache exactly like visible output — so a high-effort request’s cache footprint grows monotonically for the entire duration of its thinking.

That has a concrete effect on achievable batch size. When average sequence length rises, fewer sequences fit in the same GPU memory, batch size falls, and per-token throughput falls with it. The pathological case is a mixed pool: short chat requests interleaved with long reasoning requests, where the long ones progressively squeeze out the short ones and preemption or recompute kicks in. This is the strongest practical argument for effort-segregated pools rather than one undifferentiated fleet.

Prompt caching interacts badly too. Anthropic documents that changes to thinking parameters — enabling, disabling, or changing budget allocation — invalidate message cache breakpoints, though cached system prompts and tool definitions survive. So a router that varies effort per request across an otherwise identical conversation prefix is quietly destroying cache hits it could have kept. If you are going to vary effort, vary it at conversation boundaries where you can, not mid-thread.

Multi-turn accounting adds another wrinkle. On Claude Opus 4.5+ and Sonnet 4.6+, thinking blocks from previous turns are kept in context by default and count as input tokens when read from cache. Earlier Opus/Sonnet models and all Haiku models strip them. Same API, opposite context-growth behaviour, depending on which model string you passed.

Building a Router That Sets Effort From Request Features

The routing problem here is a close cousin of model routing, and the same architecture applies: a cheap classifier in front, a policy layer, a dispatch layer. If you have already built the semantic router pattern for inference routing, effort routing slots into the same seam — you are choosing a parameter instead of an endpoint.

Figure 2: An effort router combines a task-class prediction, an SLO-aware downgrade gate, and a verifier-driven escalation path.

Three ideas in that diagram earn their place. First, the classifier must be cheap — its own latency is pure overhead on every request, so an embedding-plus-logistic-regression model or a small fine-tuned encoder beats calling an LLM to decide how much LLM to use. Second, the SLO gate is a downgrade path: under queue pressure you shed reasoning depth before you shed requests. Third, escalation is verifier-driven rather than confidence-driven, because a model’s self-reported confidence at low effort is exactly the signal you should not trust.

Here is a working sketch of the policy layer, written against the OpenAI Responses API with a normalisation shim:

import os, time, re
from dataclasses import dataclass
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

EFFORT_LADDER = ["minimal", "low", "medium", "high"]

@dataclass
class RoutingDecision:
    effort: str
    max_output_tokens: int
    reason: str

def classify(prompt: str) -> str:
    """Stand-in for a trained classifier. Replace with an encoder model."""
    p = prompt.lower()
    if re.search(r"\b(prove|derive|debug|refactor|migrate|reconcile)\b", p):
        return "deep"
    if re.search(r"\b(summari[sz]e|extract|classify|format|translate)\b", p):
        return "shallow"
    return "moderate"

CLASS_TO_EFFORT = {"shallow": "minimal", "moderate": "medium", "deep": "high"}

# Reserve headroom so a truncated thought is not the default outcome.
CLASS_TO_CAP = {"shallow": 2_000, "moderate": 16_000, "deep": 48_000}

def decide(prompt: str, queue_depth: int, slo_ms: int) -> RoutingDecision:
    task_class = classify(prompt)
    effort = CLASS_TO_EFFORT[task_class]
    cap = CLASS_TO_CAP[task_class]
    reason = f"class={task_class}"

    # SLO-aware downgrade: shed depth before shedding requests.
    if queue_depth > 64 or slo_ms < 3_000:
        idx = max(0, EFFORT_LADDER.index(effort) - 1)
        effort = EFFORT_LADDER[idx]
        cap = min(cap, 8_000)
        reason += ";downgraded_for_load"

    return RoutingDecision(effort=effort, max_output_tokens=cap, reason=reason)

def run(prompt: str, decision: RoutingDecision):
    t0 = time.perf_counter()
    resp = client.responses.create(
        model="gpt-5.5",
        input=prompt,
        reasoning={"effort": decision.effort, "summary": "auto"},
        max_output_tokens=decision.max_output_tokens,
    )
    elapsed_ms = (time.perf_counter() - t0) * 1000

    usage = resp.usage
    reasoning_tokens = usage.output_tokens_details.reasoning_tokens

    telemetry = {
        "effort": decision.effort,
        "routing_reason": decision.reason,
        "status": resp.status,
        "input_tokens": usage.input_tokens,
        "output_tokens": usage.output_tokens,
        "reasoning_tokens": reasoning_tokens,
        "visible_tokens": usage.output_tokens - reasoning_tokens,
        "latency_ms": round(elapsed_ms, 1),
        "truncated": resp.status == "incomplete",
    }
    return resp, telemetry

Note what the telemetry dict captures. reasoning_tokens comes from usage.output_tokens_details.reasoning_tokens, and subtracting it from output_tokens gives the visible portion. That ratio — reasoning tokens over visible tokens — is the single most useful derived metric in the whole system, and almost nobody logs it.

Escalation beats over-provisioning

The instinct when quality matters is to set high effort everywhere and accept the cost. Escalation is usually strictly better, and the arithmetic is easy to see.

Suppose 80% of requests are correctly answered at low effort, and you have a verifier — unit tests, a schema check, a retrieval-grounding assertion, a cheap judge model — that catches most of the failures. Running everything at low effort and escalating the 20% that fail costs roughly 0.8 × low + 0.2 × (low + high), which is low + 0.2 × high. Running everything at high effort costs high. Whenever high effort is more than about 1.25× the cost of low effort, escalation wins. Given that high effort routinely produces an order of magnitude more reasoning tokens than low effort, the margin is enormous.

The catch is that escalation adds a full round trip to the tail. Requests that escalate pay both generations sequentially. So escalation is right for asynchronous and agentic workloads, and wrong for anything with a hard interactive deadline where you would rather burn tokens than blow the SLO. This is the same trade-off as speculative execution, and it deserves the same treatment: measure the escalation rate, and if it exceeds roughly a third, your base tier is set too low.

Features that actually predict required effort

In practice the useful signals are boringly structural. Presence of an explicit multi-step instruction. Number of distinct entities or constraints in the prompt. Whether the request is a transformation of supplied text (rarely needs thinking) or a derivation from it (often does). Whether a tool loop is involved. Conversation depth, since later turns in an agentic trajectory tend to be harder. Tenant tier, which is a policy input rather than a difficulty signal but belongs in the same decision.

What does not work well: prompt length alone, keyword lists that pattern-match on words like “analyse”, and user self-reported urgency. Prompt length correlates with input size, not reasoning difficulty — a 40-token combinatorics question needs far more thinking than a 4,000-token document summary.

Budget Enforcement and What Happens Mid-Thought

Every effort-control system eventually hits the case the docs gloss over: the budget runs out while the model is still reasoning. The semantics differ sharply between hosted APIs and self-hosted engines, and getting this wrong produces silent quality regressions.

Figure 3: The engine counts reasoning tokens from the start marker and, on exhaustion, forces the reasoning-end marker so the model transitions to answering.

On the OpenAI Responses API, exceeding max_output_tokens returns a response with status of incomplete and incomplete_details.reason set to max_output_tokens. The documentation is explicit about the worst case: this can occur before any visible output tokens are produced, so you are billed for input and reasoning tokens and receive no answer. That is the failure mode to instrument. OpenAI recommends reserving at least 25,000 tokens for reasoning and output when starting out, and tightening the buffer once you know your workload’s real reasoning-token distribution.

Anthropic’s behaviour differs by generation. max_tokens includes the thinking budget and is enforced as a strict limit. On Claude 4.5 models and newer, if input plus max_tokens exceeds the context window the API accepts the request, and generation that reaches the context limit stops with stop_reason: "model_context_window_exceeded" — earlier models rejected the request up front with a validation error instead. Any code that branches on stop reasons needs to handle both.

The self-hosted case is the most interesting, because vLLM exposes the enforcement mechanism directly rather than hiding it. Reasoning-token counting begins at reasoning_start_str; once the count reaches thinking_token_budget, vLLM forces emission of reasoning_end_str, terminating the reasoning block and pushing the model into its answer:

vllm serve Qwen/Qwen3-0.6B \
  --reasoning-parser qwen3 \
  --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": "I have to give the solution based on the reasoning directly now.</think>"}'

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "9.11 and 9.8, which is greater?"}],
    "thinking_token_budget": 1024
  }'

The reasoning_end_str in that example is doing something subtle and clever. Rather than jamming a bare </think> token into a half-finished thought, it prepends a transition phrase — “I have to give the solution based on the reasoning directly now.” — so the model sees a natural handoff rather than an abrupt amputation. vLLM’s documentation notes this explicitly as a way to make budget-exhausted termination more natural.

That design detail generalises into a principle. Truncation is not a neutral operation. A model whose reasoning is cut off mid-derivation is being asked to answer from an incomplete scratchpad, and it will usually comply rather than refuse. The result is a confident answer built on partial reasoning — which is a worse outcome than either a complete cheap answer or an explicit failure. Instrument budget-exhaustion rate as a first-class quality metric, not just a cost metric.

The rest of vLLM’s surface is worth knowing precisely. --reasoning-parser selects the extraction logic (qwen3, deepseek_r1, glm45, gemma4, granite, minimax_m2_append_think and others). Parsed reasoning arrives in a reasoning field on the message — renamed from reasoning_content, so older client code needs updating. --default-chat-template-kwargs '{"enable_thinking": false}' sets a server-wide default that per-request chat_template_kwargs can override. And usefully, setting reasoning_effort to low, medium or high now auto-injects enable_thinking: true, while none injects false — so the OpenAI-shaped parameter works against models that natively need a chat-template flag.

Inkling shipped with inference support across vLLM, SGLang, TokenSpeed and llama.cpp through partner contributions. Because budget-enforcement semantics vary by engine and version, verify the exact behaviour on the version you deploy rather than assuming parity across the ecosystem.

Why Effort Control Changes Unit Economics

Reasoning tokens bill as output tokens on every major platform, and output tokens are the expensive side of the ledger. That single fact restructures the cost model of an LLM product.

Under the old model, cost per request was roughly proportional to visible output length, which was itself bounded by product design — a chat reply is a few hundred tokens because that is what fits on a screen. Under reasoning models, cost per request is dominated by an invisible quantity you do not control directly and cannot see in the response body. Two users asking superficially similar questions can differ tenfold in cost.

Worse, the invisible portion is not proportional to anything the user perceives. You are billed for full thinking tokens even when you never see them. Anthropic bills the original thinking tokens whether display is summarized or omitted, and explicitly warns that the billed output-token count will not match the visible count. Google’s docs say the same thing for Gemini: pricing is based on full thought tokens even though only summaries are returned. OpenAI likewise does not expose raw reasoning tokens, only optional summaries via reasoning.summary, while billing the underlying reasoning at the output rate.

The practical implication is that per-request cost variance explodes, and any pricing model built on average cost per request becomes unsafe. Seat-based pricing with a heavy-user tail, or credit systems denominated in requests rather than tokens, will bleed margin on exactly the customers who use the product most seriously.

A worked example makes the shape visible. The following numbers are illustrative, not vendor-quoted — plug in your own rates and measured token counts.

Tier	Reasoning tokens	Visible tokens	Total output tokens	Relative cost
Minimal	0	400	400	1.0×
Low	1,200	400	1,600	4.0×
Medium	6,000	500	6,500	16.3×
High	30,000	600	30,600	76.5×

Illustrative only. The point is the ratio, not the absolute figures.

Two things fall out. First, the span between cheapest and most expensive tier on the same model here is roughly 76×, which is wider than the price gap between most vendors’ small and flagship models. Effort control gives you more cost leverage than model selection does. Second, visible output barely moves across tiers — so from a user’s perspective, a 76× cost difference produces a response of nearly identical length. There is no natural feedback signal telling anyone that the expensive path was taken.

This is also where the Inkling result becomes economically meaningful rather than merely interesting. If one model reaches a given score at a third of another’s token count, its effective cost per solved task is a third, regardless of headline per-token pricing. Evaluating models on price-per-million-tokens without normalising for tokens-per-solved-task is now a category error. The right comparison is cost per unit of completed work, which is the framing we develop in inference cost optimisation for 2026.

One more accounting subtlety: for stateless deployments on the Responses API, preserving reasoning across turns requires adding reasoning.encrypted_content to the include parameter, after which reasoning items carry an encrypted_content property you pass back on subsequent turns. Those returned items become input tokens on the next call. Reasoning you paid for as output once becomes input you pay for again on every following turn — a cost that grows with conversation depth and is invisible unless you are reconciling usage per turn.

Does More Thinking Actually Help Your Task?

The assumption underneath every effort-control deployment is that quality increases monotonically with effort. For many tasks it does. For a meaningful minority it does not, and the failure is not subtle once you look for it.

The general shape is an inverse-scaling regime: on certain task types, extended reasoning degrades accuracy relative to a direct answer. The mechanisms are recognisable to anyone who has watched a model think aloud. On simple factual retrieval, extended reasoning gives the model room to talk itself out of a correct initial instinct. On tasks with a strong prior, deliberation surfaces plausible-sounding alternatives and the model may select one. On constraint-satisfaction problems with a unique answer, long traces accumulate arithmetic or bookkeeping slips that a short trace never had the opportunity to make.

There is also a pure-overthinking regime distinct from outright inversion: accuracy plateaus while tokens keep climbing. This is more common and more expensive than true inverse scaling, and it is what an effort sweep is designed to find.

The empirical procedure is straightforward and every team deploying effort control should run it before choosing defaults:

import json, statistics
from collections import defaultdict

EFFORTS = ["minimal", "low", "medium", "high"]

def sweep(eval_set, run_fn, n_repeats=3):
    """Run the full eval set at each effort level; return score and token cost."""
    results = defaultdict(lambda: {"scores": [], "reasoning_tokens": []})
    for effort in EFFORTS:
        for item in eval_set:
            for _ in range(n_repeats):
                out, telem = run_fn(item["prompt"], effort)
                results[effort]["scores"].append(item["grade"](out))
                results[effort]["reasoning_tokens"].append(telem["reasoning_tokens"])
    summary = {}
    for effort, r in results.items():
        summary[effort] = {
            "accuracy": statistics.mean(r["scores"]),
            "mean_reasoning_tokens": statistics.mean(r["reasoning_tokens"]),
            "p95_reasoning_tokens": sorted(r["reasoning_tokens"])[int(0.95 * len(r["reasoning_tokens"]))],
        }
    return summary

def knee_point(summary, tolerance=0.01):
    """Lowest effort whose accuracy is within `tolerance` of the best observed."""
    best = max(s["accuracy"] for s in summary.values())
    for effort in EFFORTS:
        if summary[effort]["accuracy"] >= best - tolerance:
            return effort
    return EFFORTS[-1]

print(json.dumps(sweep(my_eval_set, run_with_effort), indent=2))

Three methodological points matter more than the code. Repeat each item several times, because reasoning-model outputs are high-variance and a single sample per effort level will produce noise you will misread as signal. Segment the results by task class rather than reporting one aggregate number — the whole reason to build a router is that different classes have different knee points, and averaging hides exactly the structure you need. And record the token distribution, not just the mean, because p95 reasoning tokens drives your capacity planning while the mean drives your bill.

The output of this sweep is the router’s configuration. For each task class you get a knee point — the effort level beyond which accuracy stops improving — and you set that as the class default. Classes whose curve is flat from minimal upward should never be routed above minimal, and finding even one such class in your workload usually pays for the entire exercise.

Note that Inkling’s own release illustrates why the curve is the right object of study rather than a single score. Sweeping effort from 0.2 to 0.99 traces performance against mean generated tokens on Terminal Bench 2.1, HLE and IFBench, and the vendor’s framing is that looking at the full cost curve is what lets developers choose the right model per use case. That is the correct methodology applied to model selection; the same methodology applied to your own eval set is what should set your defaults.

Observability of Reasoning Tokens

You cannot manage what you cannot see, and reasoning tokens are designed to be partially invisible. The observability requirements are therefore unusually specific.

Figure 4: Reasoning-token telemetry splits into metrics for alerting, traces for latency attribution, and a per-tenant ledger for unit economics.

The fields to capture exist on every major API and are consistently under-collected. OpenAI exposes usage.output_tokens_details.reasoning_tokens. Anthropic exposes usage.output_tokens_details.thinking_tokens, which the docs describe as a read-only observability breakdown while output_tokens remains the authoritative billing total — and note that when streaming, this breakdown appears only on the final message_delta event. Gemini’s Interactions API reports usage.total_thought_tokens. vLLM returns parsed reasoning in the reasoning field, from which you can count directly.

Log all of them as histograms rather than counters, tagged with the effort level, the routing reason, the task class and the tenant. Averages are useless here; the whole story is in the tail.

Four alerts earn their keep. Effort drift: the distribution of assigned effort shifting without a deploy, which usually means input mix changed or your classifier is degrading. Budget-exhaustion rate: the fraction of requests hitting incomplete, model_context_window_exceeded, or a forced reasoning-end marker, which is a quality alarm dressed as a cost metric. Reasoning-to-visible ratio: a rising ratio at fixed effort means the model is working harder for the same output, worth investigating before the invoice arrives. Escalation rate: rising escalation means your base tier no longer matches the workload.

For tracing, the effort level and realised reasoning-token count belong as span attributes on the model call. Without them, tail-latency investigations dead-end at “the model was slow” instead of reaching “this cohort of requests was routed to high effort and produced 30k reasoning tokens each.” Both Anthropic and Gemini offer thinking summaries, and while summaries are not a substitute for traces, sampling them on slow or failed requests is a genuinely useful debugging aid.

Trade-offs, Gotchas, and What Goes Wrong

Effort settings are not portable. A migration script that maps high to budget_tokens: 32000 is guessing, and the guess will be wrong in both directions across model families. Re-run your sweep per model rather than porting configuration.

Feature incompatibilities are silent traps. Anthropic’s extended thinking is incompatible with temperature and top_k modifications and with forced tool use; top_p is constrained to between 0.95 and 1; and you cannot pre-fill responses. Tool use with thinking supports only tool_choice: {"type": "auto"} or {"type": "none"} — passing any or a named tool errors out. Teams that relied on forced tool calls for schema guarantees have to rearchitect.

Toggling thinking mid-turn fails quietly. Anthropic’s docs describe graceful degradation: if a mid-turn conflict occurs, the API disables thinking for that request rather than erroring. Your request succeeds, your quality drops, and nothing in the response tells you unless you check for the presence of thinking blocks.

Dropping redacted_thinking blocks breaks multi-turn. Application code that filters content blocks with block.type == "thinking" silently discards redacted_thinking blocks, producing a 400 on the next turn. The same class of bug exists in Gemini’s stateless mode, where thought signatures must be resent unmodified.

Cache invalidation from effort variation. Changing thinking parameters invalidates message cache breakpoints. A router that varies effort turn-by-turn within one conversation can destroy more value in lost cache hits than it saves in reasoning tokens.

Long budgets meet infrastructure limits. Anthropic’s SDKs require streaming when max_tokens exceeds 21,333 to avoid HTTP timeouts, and recommend batch processing above 32k thinking tokens because long-running requests hit system timeouts and connection limits. Load balancers, ingress controllers and API gateways in front of your service have their own idle timeouts that were sized for sub-minute responses.

Reasoning traces are not an audit log. Summaries are generated by a separate model in Anthropic’s case, and raw traces are unavailable on several frontier models by policy. Compliance requirements that assume you can produce the model’s actual reasoning are unsatisfiable on those endpoints.

Practical Recommendations

Start by measuring before you configure. Run an effort sweep on a representative eval set segmented by task class, find the knee point for each, and set those as defaults. Most teams discover at least one high-volume class that gains nothing above minimal effort, and that discovery alone typically justifies the work.

Separate the policy decision from the placement decision. Let a cheap classifier assign an effort tier from request features, and let capacity management decide which pool serves it. Segregate pools by budget class so long traces cannot starve short requests through KV-cache pressure.

Prefer escalation over over-provisioning wherever a verifier exists and the workload tolerates an extra round trip. Set the base tier so escalation runs below roughly a third of requests, and treat a rising escalation rate as a signal to re-tune rather than a steady state.

Treat truncation as a quality event. A budget-exhausted request that returns a confident answer from a half-finished thought is worse than a cheap complete answer, so alert on exhaustion rate and reserve real headroom above your measured p95 reasoning tokens.

A short checklist before shipping effort control:

Effort sweep run per task class, with repeats, and knee points recorded
Reasoning-token counts logged as tagged histograms, not averages
Budget-exhaustion and incomplete responses alerted on
Reasoning-to-visible token ratio tracked per tenant and per effort tier
p95 reasoning tokens used for capacity planning, mean used for cost forecasting
SLO expressed in time-to-first-visible-token
Multi-turn round-tripping of thinking blocks and signatures verified, including redacted blocks
Timeouts on every network hop checked against your longest expected high-effort request
Cache-invalidation impact of per-request effort variation measured before rollout

Frequently Asked Questions

What is reasoning effort control in LLM serving?

Reasoning effort control is the per-request ability to set how many tokens a model spends on internal reasoning before producing a visible answer. It is exposed as categorical levels such as low, medium and high, as integer token budgets, or as a continuous scalar. Because the setting changes generation length rather than weights, it moves latency, cost and quality along a single axis on one deployed model, turning what used to be a model-selection decision into a runtime policy.

How is a thinking budget different from max_tokens?

A thinking budget bounds only the internal reasoning portion, while max_tokens or max_output_tokens bounds reasoning plus visible output together. Anthropic requires budget_tokens to be less than max_tokens in standard extended thinking, though interleaved thinking with tools relaxes this since the budget then spans an entire assistant turn. In vLLM, thinking_token_budget is a separate sampling parameter enforced independently of max_tokens.

What happens when a model runs out of thinking budget mid-thought?

Behaviour differs by platform. OpenAI returns status: "incomplete" with incomplete_details.reason set to max_output_tokens, and this can happen before any visible output exists — so you are billed with nothing to show. Anthropic stops with stop_reason: "model_context_window_exceeded" on 4.5-generation models and newer. vLLM forces emission of reasoning_end_str, optionally preceded by a transition phrase, pushing the model into answering from an incomplete scratchpad.

Does higher reasoning effort always improve accuracy?

No. Some task types show inverse scaling, where extended reasoning degrades accuracy — typically simple factual retrieval, tasks with strong correct priors, and constraint problems where long traces accumulate bookkeeping errors. More common is a plateau where accuracy stops improving while token spend keeps climbing. The only reliable way to know is an effort sweep on your own eval set, segmented by task class and repeated several times per item.

Are reasoning tokens billed even when they are hidden?

Yes, on every major platform. Anthropic bills the full thinking tokens regardless of whether display is summarized or omitted, and warns that billed output tokens will not match visible tokens. Google bills full thought tokens even though only summaries are returned. OpenAI bills reasoning at the output rate while exposing only optional summaries. Read output_tokens_details.reasoning_tokens, output_tokens_details.thinking_tokens, or total_thought_tokens respectively to see the real spend.

How do open-weight models expose thinking control?

Qwen3 uses a hard switch via chat_template_kwargs: {"enable_thinking": false} plus soft /think and /no_think prompt markers. Thinking Machines Lab’s Inkling, released July 2026 under Apache 2.0, exposes a continuous effort dial swept from 0.2 to 0.99 in its release evaluations, with named reasoning_effort levels in transformers. In vLLM, --reasoning-parser handles extraction and setting reasoning_effort auto-injects enable_thinking for models that require it.

Reasoning-Effort Control in LLM Serving: Thinking Budgets (2026)