LLM Inference SLO and Goodput: TTFT, TPOT and Capacity Planning for Production Serving

Most teams size an LLM deployment by running a benchmark until the GPUs saturate and then quoting tokens per second. That number is nearly useless for a product. A server that emits 20,000 tokens per second while half of its users wait four seconds for the first word is failing, and the throughput dashboard will still look green. The metric that survives contact with users is an llm inference slo goodput target: the highest request rate a replica can sustain while a stated fraction of requests still meets its latency objectives.

This matters more in 2026 than it did two years ago. Reasoning models emit thousands of hidden tokens, agent loops chain dozens of calls, and prefix caches, chunked prefill and prefill/decode disaggregation all change where latency comes from. Every one of those features trades one latency component against another, so you cannot judge them without per-component objectives.

You will leave with precise definitions of time to first token (TTFT), time per output token (TPOT) and inter-token latency (ITL), a method for choosing SLOs per product surface, a queueing model of prefill and decode, the percentile mistakes that corrupt dashboards, a fully worked and clearly labelled illustrative capacity plan, a load-testing recipe, and burn-rate alerting.

What this covers: the metric vocabulary, goodput as the capacity unit, SLO design per surface, queueing and percentile math, a worked sizing example, honest load testing, tooling, and the failure modes that break plans in production.

Context and Background

Classic web SLOs revolve around one number: request latency at some percentile. LLM serving breaks that model because a single request has two physically different phases. Prefill processes the whole prompt in parallel and is compute-bound; it produces the first token. Decode generates the remaining tokens one step at a time and is dominated by memory bandwidth, because each step reads the model weights and the growing key-value (KV) cache. A request’s latency is therefore a sum of two quantities that respond to load in opposite ways.

The vocabulary matters because tools define it differently. The DistServe paper (OSDI 2024) defines TTFT as the duration of the prefill phase and TPOT as the average time to generate each token excluding the first, so total latency equals TTFT plus TPOT multiplied by the number of generated tokens. The vLLM benchmarking documentation uses the same per-request TPOT, computed as end-to-end latency minus TTFT, divided by output tokens minus one. ITL is different: it is the gap between consecutive streamed outputs, aggregated across all requests rather than averaged within one. Mixing these up is the most common reason two teams report different “inter-token latency” for the same server.

Throughput-first benchmarking grew up in an offline world. Batch jobs want maximum tokens per GPU-hour and do not care about tail latency; our post on offline batch LLM inference covers when that framing is correct. Interactive serving is the opposite regime. Pushing batch size up raises throughput and also raises per-step decode time, so TPOT degrades while the throughput chart improves.

The concept that reconciles the two views is goodput. DistServe defines it as the maximum request rate that can be served while meeting the SLO attainment goal, for example 90 percent, per GPU provisioned. The authors report serving up to 7.4 times more requests or meeting up to 12.6 times tighter SLOs than state-of-the-art baselines within their evaluation, while staying inside latency constraints for more than 90 percent of requests. Those are the paper’s results on its own workloads and hardware; treat them as evidence the framing is powerful, not as a number your cluster will reproduce. Throughout this post, “goodput” means that SLO-conditioned rate, not the looser use of the word as “useful tokens”.

If you have already read our comparison of engines in vLLM vs SGLang vs TensorRT-LLM, the practical upshot is that engine choice moves your goodput curve, but the SLO definition decides which point on the curve you operate at.

The Reference Model: Metrics, SLOs and Goodput

Direct answer: An LLM inference SLO is a pair of per-request latency objectives, one for TTFT and one for TPOT, plus an attainment target such as 95 percent. Goodput is the highest request rate per GPU at which that attainment target still holds. You size capacity from goodput, never from peak throughput.

Figure 1: Where each latency metric sits on a single request timeline, from arrival through queue, prefill and decode.

Figure 1 decomposes one request. Time starts when the client sends the request, not when the engine accepts it, which matters because queue wait inside a gateway or scheduler is invisible to engine-side metrics. The segments are queue wait, prefill, the first-token emission that ends TTFT, and then a sequence of decode steps whose average spacing is TPOT. End-to-end latency (E2E) is everything from send to final token. The long-description version: TTFT is the user’s perceived “start” delay, TPOT is the perceived “speed” of the stream, and E2E is the total wait for a non-streaming consumer.

Definitions you should pin in writing

Write the following into your SLO document, because ambiguity here causes months of argument later.

TTFT: client send time to arrival of the first output token chunk. Includes queueing, tokenization, prefill, and network. State whether you measure at the client, the gateway or the engine.
TPOT: per-request mean of (E2E minus TTFT) divided by (output tokens minus one). Hides single long stalls inside a stream, because a three-second freeze in a 500-token reply barely moves the mean.
ITL: distribution of individual gaps between streamed chunks. Exposes stalls. With speculative decoding several tokens can arrive in one chunk, so ITL no longer equals per-token time; the vLLM documentation notes it does not record zero-length gaps for those.
E2E latency: the figure that matters for tool calls, agents and any consumer that waits for the full response.
SLO attainment: the fraction of requests meeting every objective simultaneously. A request that hits TTFT but misses TPOT is a miss.

Why goodput, and why per GPU

Goodput has two properties that throughput lacks. It is conditioned on user experience, so it cannot be inflated by letting latency balloon. And it is expressed per provisioned GPU, so it translates directly into cost per query: dollars per GPU-hour divided by (goodput times 3,600) gives dollars per compliant request. That is the unit a finance team can compare across engines, quantizations and hardware generations.

A subtle point: goodput depends on the request mix. Doubling average prompt length changes prefill cost roughly in proportion and changes KV cache pressure, so a goodput figure is only meaningful alongside the input and output length distributions that produced it. A goodput number without its workload description is as unusable as a throughput number without its SLO.

The two-phase view of load

Prefill and decode saturate differently. Prefill throughput is limited by compute; adding more prompt tokens to a step lengthens that step nearly linearly once the GPU is saturated. Decode is limited by memory bandwidth and KV cache capacity: each step’s cost is only mildly sensitive to batch size until the batch is large, which is why batching decode is so profitable for throughput and so dangerous for tail latency once the step time creeps up.

When both phases share one batch, DistServe calls the resulting slowdown prefill-decoding interference. Adding a single prefill job to a batch of decoding requests slows both, and the paper concludes that colocated batching “invariably leads to compromises in either TTFT or TPOT”. Chunked prefill is the standard mitigation inside one engine, as covered in our deep dive on chunked prefill and prefill/decode interference. Disaggregation is the stronger one, with a Kubernetes-native implementation discussed in llm-d. Both exist because the two SLOs pull in different directions.

Setting SLOs Per Product Surface

A single global SLO is the usual mistake. A voice assistant, a code completion plug-in, a chat window and a nightly summarizer have unrelated tolerances, and forcing them onto one tier either overspends on the lax surfaces or underserves the strict ones.

Figure 2: Mapping product surfaces to TTFT and TPOT classes and then to separate serving pools.

Figure 2 shows the idea: classify traffic at the gateway, attach an SLO class to each request, and route classes to pools or priority tiers whose capacity is planned independently. The table below gives starting points. These are illustrative engineering heuristics, not published standards; calibrate them with your own user research and with how fast your users actually read or listen.

Surface	What the user perceives	Illustrative TTFT target (p95)	Illustrative TPOT target (p95)	Notes
Voice assistant	Silence, then speech	300-500 ms	20-40 ms	Downstream speech synthesis consumes tokens; TTFT dominates
Inline code completion	Ghost text appearing	150-400 ms	Not user visible	Short outputs; E2E is the real metric
Chat window	Reply starting, then streaming	0.8-1.5 s	30-60 ms	Streaming speed above human reading speed is wasted
Agent tool call	Whole response needed	Loose	Loose	Set an E2E budget instead; stream not consumed
Document summarization	Progress bar	2-5 s	50-100 ms	Long prompts make prefill expensive
Offline batch	Job completion	None	None	Optimize throughput, see batch inference post

Choose TPOT from human reading speed, not from hardware

A common error is to set TPOT to whatever the hardware delivers. Adult silent reading speed is often cited around 200 to 300 words per minute, which is on the order of 5 to 8 tokens per second for English text at roughly 1.3 tokens per word. That implies a TPOT near 125 to 200 ms would still keep pace with a reader. Teams set tighter targets, 30 to 60 ms, because streaming feels sluggish well before it falls behind a reader and because token counts per word vary by language and tokenizer. The point is that the target is a product decision with a defensible upper bound, not a number to copy from a competitor. Treat the reading-speed arithmetic as a rough sanity check, since reading speeds vary widely by person and content.

Reasoning models change the arithmetic

With models that think before answering, TTFT can mean “first visible token” or “first token of any kind”, and the two differ by the entire reasoning trace. Decide explicitly. Many products stream a “thinking” indicator and define the SLO from the first visible answer token, which puts the reasoning duration inside TTFT and makes it long-tailed. Others cap thinking with an explicit token budget; the mechanics of that control are covered in reasoning effort control and thinking budgets. If you do not separate hidden-token time from visible-token time in your metrics, your TTFT percentile will mostly measure how hard users’ questions were.

Attach an attainment target and a window

An SLO is not complete until it says what fraction of requests, over what window. A compact specification looks like this:

slo_class: chat_interactive
objectives:
  ttft_seconds: {threshold: 1.2, attainment: 0.95}
  tpot_seconds: {threshold: 0.05, attainment: 0.95}
joint_attainment_target: 0.92   # both must hold on the same request
window: 28d
workload_assumption:
  input_tokens: {p50: 900, p95: 6000}
  output_tokens: {p50: 350, p95: 1500}

Note the joint target. If TTFT and TPOT attainment are each 95 percent and the failures never overlap, joint attainment is as low as 90 percent; if they overlap perfectly it is 95 percent. Because prefill-heavy requests tend to be slow on both axes, overlap is real but unknowable without measuring the joint event. Always compute attainment per request, then aggregate.

Queueing, Percentile Math and the Capacity Model

This section is where most plans quietly go wrong. Three ideas do the work: a queueing view of prefill, Little’s law for decode concurrency, and honest percentile handling.

Figure 3: Requests wait in a prefill queue, then occupy a decode slot for their whole generation; each stage has its own saturation behavior.

Prefill behaves like a queue with a nearly deterministic server

When prefill runs on its own device pool, as in a disaggregated deployment, execution time for a given prompt length is almost deterministic, and arrivals from many independent users are close to Poisson. The DistServe paper models this as an M/D/1 queue. With execution time D per request, arrival rate R and utilization RD below one, the average TTFT for a single device is:

Avg_TTFT = D + R*D^2 / (2*(1 - R*D))

The first term is service time and the second is queueing delay. The key property is the denominator: delay does not grow linearly with load, it blows up as utilization approaches one.

A worked check, with numbers chosen for illustration only. Suppose prefill for the typical prompt takes D = 0.4 s.

At R = 2.0 requests/s, utilization is 0.8. Queue delay is 2.0 x 0.16 / (2 x 0.2) = 0.8 s, so average TTFT is 1.2 s.
At R = 2.25 requests/s, utilization is 0.9. Queue delay is 2.25 x 0.16 / (2 x 0.1) = 1.8 s, so average TTFT is 2.2 s.
Increase load by 12.5 percent and mean TTFT rises by 83 percent. The p95 rises faster than the mean.

This is why a utilization ceiling is part of the SLO design rather than a comfort margin. For the general theory and a version tuned to platform workloads, see our post on queueing theory, Little’s law and tail latency. The M/D/1 result is a model for the prefill pool in isolation. In a colocated engine, prefill chunks interleave with decode steps and the real system is messier; use the formula to build intuition and to reason about disaggregated pools, not as a prediction for a colocated vLLM instance.

DistServe also gives variants for parallelism: with two-way inter-operator parallelism the delay term becomes RD^2 / (4(2 – RD)), and with two-way intra-operator parallelism and speedup coefficient K it becomes D/K + RD^2 / (2K(K – R*D)). The paper’s conclusion is that intra-operator parallelism helps most at low arrival rates, where execution time dominates, while inter-operator parallelism wins as rates rise and queueing dominates. The practical lesson is that the right parallelism for the prefill pool depends on where you sit on the load curve, which is another reason to plan from goodput.

Decode is governed by Little’s law and memory

Little’s law says the average number of requests in a system equals arrival rate times average time in the system. For decode, that is the number of concurrent sequences:

concurrent_sequences = arrival_rate x mean_decode_time

with mean_decode_time = mean_output_tokens x mean_TPOT. Each concurrent sequence holds KV cache memory proportional to its current length. So the real ceiling on a replica is the minimum of two limits: the batch size at which step time stays under your TPOT target, and the number of sequences whose KV cache fits in GPU memory after weights are loaded. Which limit binds first depends on model size, context length, KV precision and attention variant; our post on multi-head latent attention versus GQA and MQA explains why architecture changes this ceiling so much.

Percentile mistakes that corrupt dashboards

Four errors appear again and again.

Averaging percentiles across replicas. The mean of per-replica p95s is not the fleet p95. Aggregate histogram buckets first, then compute the quantile. With Prometheus histograms that means summing bucket counters across instances before applying a quantile function.
Using the mean TPOT as a stall detector. Per-request mean TPOT smooths out freezes. Track ITL p99 as a second signal, or a stall-count metric, if streaming smoothness matters.
Reading a high percentile from too few samples. For a p99 estimate you need enough requests that the tail contains more than a handful of points. As a rule of thumb, expect at least about 1,000 samples per cell to look at p99 and treat p99.9 as unmeasurable in a short test. The standard error of an attainment fraction p estimated from n samples is roughly sqrt(p(1-p)/n); at p = 0.95 and n = 200 that is about 1.5 percentage points, wide enough to mistake 95 percent for 92 percent.
Conditioning on completed requests. Timed-out and dropped requests have no TTFT. If you compute percentiles only over successes, an overloaded system improves its own statistics. Count failures and timeouts as SLO misses.

Engine metrics you can actually scrape

vLLM exposes Prometheus histograms including vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:request_prefill_time_seconds and vllm:request_decode_time_seconds, plus gauges for running and waiting requests and KV cache usage. It also has a per-request vllm:request_time_per_output_token_seconds histogram. Metric names change between releases, so confirm them against your pinned version’s documentation. For tracing request flow across gateway, router and engine, the OpenTelemetry work covered in our GenAI semantic conventions post gives a vendor-neutral vocabulary.

Engine-side metrics start the clock when the engine receives the request. Add a client-side or gateway-side measurement, because gateway queueing, retries and TLS are part of what users feel. The vLLM benchmark client makes the same point: it has an option to report time spent waiting for a client slot when you cap concurrency.

A Worked Capacity Plan (Illustrative Numbers)

Every figure in this section is invented for the exercise. It shows the method; your own sweep will produce different inputs. Do not quote these as benchmarks.

Figure 4: The capacity planning loop: forecast, measure goodput per replica, apply headroom and redundancy, then validate by replay.

Step 1: state the workload and the SLO

Assume a chat product with the SLO class above: TTFT p95 at most 1.2 s, TPOT p95 at most 50 ms, joint attainment target 92 percent. Assume the forecast peak is 30 requests per second, with a mean input of 1,200 tokens and a mean output of 400 tokens. (Peak means the busiest sustained five-minute window, not an instantaneous spike.)

Step 2: derive the concurrency the decode side must hold

If the mean TPOT at the operating point is 35 ms, mean decode time is 400 x 0.035 = 14 s. Add a 0.5 s mean TTFT and E2E is about 14.5 s. Little’s law gives 30 x 14.5 = 435 concurrent sequences fleet-wide at peak. If each sequence averages 1,200 + 200 = 1,400 tokens of live context (the input plus on average half the output), the fleet must hold 435 x 1,400, about 609,000 tokens of KV cache. Whether that fits depends on bytes per token, which depends on the model; compute it from your model’s layer count, KV heads, head dimension and KV precision rather than guessing.

Step 3: measure goodput per replica, do not compute it

Run a load sweep (next section) on one replica of your real configuration, with a length distribution matching production. Suppose the sweep shows this, again illustratively:

Offered rate (req/s)	TTFT p95	TPOT p95	Joint attainment
4	0.6 s	38 ms	99%
6	0.8 s	44 ms	97%
8	1.1 s	49 ms	93%
9	1.7 s	58 ms	71%
10	3.4 s	81 ms	38%

With a 92 percent joint target, the goodput of one replica is 8 requests/s. Notice the cliff: from 8 to 10 requests/s throughput rises 25 percent while attainment collapses. A throughput-only benchmark would have reported 10 as the winner.

Step 4: apply headroom and redundancy

Operating at exactly the cliff edge is fragile, so plan to a utilization fraction of goodput. Using 70 percent of 8, each replica is planned for 5.6 requests/s. Then:

Replicas for peak: 30 / 5.6 = 5.36, so 6.
Add N+1 so one replica can fail or drain during a rollout: 7 replicas.
Cost per compliant request at the planned operating point is the hourly GPU price times the replica’s GPU count, divided by (5.6 x 3,600). Plug in your own price.

The 70 percent factor is a judgement. Bursty traffic, long-tail prompts and slow autoscaling all argue for a lower factor; steady, well-shaped traffic with fast scale-out argues for higher. The M/D/1 numbers earlier show why: the cost of running hot grows non-linearly.

Step 5: add the dimensions that break simple plans

Prefix caching. A high cache hit rate cuts prefill work, which raises goodput, but the hit rate varies with traffic. Plan on the measured hit rate at the 10th percentile of the day, not the average. See prefix caching with RadixAttention and vLLM for what moves that rate.
Speculative decoding. It lowers effective TPOT when acceptance is high and can hurt at high batch sizes, so re-run the sweep with it on; the trade-offs are in our speculative decoding comparison.
Multiple traffic classes. Plan each pool separately, then check whether pooling them with priority queues saves GPUs without breaking the strict class.
Model upgrades. A new model with a larger output length or heavier reasoning shifts both mean decode time and KV demand. Re-baseline before cutover.

Step 6: validate by replaying, not by arithmetic

Arithmetic produces a hypothesis. Replay a recorded production trace, with real arrival timing and length distributions, at 1.0x and 1.3x peak against the proposed fleet, and confirm that joint attainment stays above target. Include a replica-loss test mid-run.

Load Testing With Realistic Length Distributions

A goodput number is only as honest as the load that produced it. Three properties of the test decide whether it predicts production.

Length distributions. Fixed-length synthetic prompts, for example every request with 256 input and 128 output tokens, produce smooth batches that no real service sees. Production lengths are heavy-tailed: a few very long prompts dominate prefill cost and KV memory. Sample input and output lengths from a recorded distribution, or at least from a fitted lognormal with the right p50 and p95, and keep correlation between input and output length if your product has it (summaries are long-in, short-out; drafting is the reverse).

Arrival process. Closed-loop tests, where N clients each send the next request when the previous finishes, self-throttle as latency rises and therefore hide overload. Open-loop tests send at a scheduled rate regardless of responses and expose the cliff. The vLLM benchmark client’s default request rate is infinite, which fires everything at once and measures peak throughput, not goodput; set a finite rate, which then follows a Poisson process, and use the burstiness option to model clumpier traffic.

Cache state. Repeated runs against one server can reuse prefix cache entries and inflate results. The vLLM documentation advises varying the seed, restarting the server, or using its sweep command, which resets caches between runs. Equally, a cold-cache test understates what a warm production system achieves; run both and report both.

Tooling

Two tools are worth knowing, and both are verified against their current documentation.

vllm bench serve benchmarks a running OpenAI-compatible vLLM server, reporting request and token throughput plus TTFT, TPOT and ITL statistics. It supports ShareGPT, random synthetic, custom JSONL, Hugging Face and several multimodal datasets, and controls for request rate, burstiness and maximum concurrency. Its own documentation recommends GuideLLM for production-server benchmarking.
GuideLLM (a vLLM project) is built for SLO-aware evaluation under realistic load. It offers sweep, constant, Poisson, synchronous, throughput and concurrent profiles; reports full distributions for TTFT, ITL and end-to-end latency; accepts synthetic data, Hugging Face datasets and local files; and writes JSON and CSV by default. Its sweep profile finds the performance range from one-at-a-time up to saturation, which is the raw material for a table like the one in the worked example. Install with pip install guidellm[recommended]. Flags and profile names evolve, so read the current README before scripting it.

# Illustrative invocation shape; confirm flags against your installed version.
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=sweep \
  --constraint kind=max_duration,seconds=300 \
  --data kind=synthetic_text,prompt_tokens=1200,output_tokens=400

I have not verified how GuideLLM expresses per-request SLO thresholds; its page describes SLO-driven evaluation without detailing the configuration. In practice, post-process the exported per-request records yourself: compute joint attainment at each offered rate and find the highest rate that clears your target. That computation is twenty lines of pandas and leaves no ambiguity about how attainment was defined.

Reading a sweep correctly

Plot attainment against offered rate and look for the knee. The knee, not the maximum throughput point, is your goodput. Check three things before trusting it: that the client was not the bottleneck (the vLLM benchmark can report client queue time), that the run was long enough to reach steady state after warm-up, and that error and timeout counts were included as misses. Repeat the run at least three times; if goodput varies by more than 10 percent between runs, your test is measuring noise, and the plan needs a wider safety factor.

Burn-Rate Alerting for Inference SLOs

Once the SLO exists, protect it in production with error-budget thinking from the SRE discipline. The budget is the allowed fraction of non-compliant requests: with a 92 percent joint attainment target, the budget is 8 percent of requests over the window. Our post on SLOs, error budgets and burn-rate alerting covers the general machinery; here is the LLM-specific adaptation.

Burn rate is how fast you consume budget relative to the rate that would exhaust it exactly at the end of the window. A burn rate of 1 uses the whole budget over the window; a burn rate of 10 exhausts it in a tenth of the window. The Google SRE Workbook recommends multi-window, multi-burn-rate alerts: for a 30-day SLO, page on a burn rate of 14.4 sustained over one hour (consuming 2 percent of the budget) and 6 over six hours (5 percent), and open a ticket at 1 over three days (10 percent), each paired with a shorter window so alerts clear quickly after recovery. Those parameters are from the workbook’s worked example for availability SLOs; adapt them rather than copying them blindly.

For LLM serving, define the “bad event” per request as a miss on either objective, and compute the ratio from histograms. With Prometheus buckets, the count of requests under the TTFT threshold divided by the total count gives per-objective attainment. Joint attainment cannot be derived from two separate histograms, so either emit a per-request boolean counter at the gateway (slo_met_total and slo_total, labelled by class) or accept the pessimistic bound of adding the two miss rates. The counter approach is cleaner and cheap.

Three LLM-specific considerations apply:

Alert on queue depth as a leading indicator. Burn-rate alerts are lagging. A rising vllm:num_requests_waiting gauge or request queue time warns minutes earlier, before attainment degrades. Use it for autoscaling and as a non-paging alert.
Segment by class and by prompt length bucket. A fleet-wide ratio hides the case where only long-prompt requests are failing. Add a length-bucket label with low cardinality, such as three buckets.
Exclude client-caused failures deliberately. Cancellations by users who navigated away are not server misses; invalid requests are not either. Decide the rule in advance and apply it identically in load tests.

Tie alerts to runbooks that name the levers in order of speed: shed or defer low-priority classes, reduce maximum output tokens, scale out, disable an expensive feature such as extended reasoning, and finally fail over. The incident framework in our post on incident management and error budgets shows how to attach severity levels to budget burn.

Trade-offs, Gotchas, and What Goes Wrong

Autoscaling lag is longer than you think. Starting a replica means provisioning a GPU, pulling weights that can be tens to hundreds of gigabytes, loading them and warming kernels. If that takes minutes, reactive scaling cannot protect a five-minute burst. Pre-scale on forecasts, keep warm spares, or use a lower utilization factor; do not assume Kubernetes will save you.

Disaggregation is not free. Splitting prefill and decode removes interference but adds KV cache transfer between pools, a second pool to size, and new failure modes when the transfer link is slow. The DistServe gains were measured with fast interconnects in their testbed; on a slower network the transfer cost can erase them. Our coverage of NVIDIA Dynamo and LMCache looks at the KV movement side.

Chunked prefill trades TTFT for TPOT. Smaller chunks protect decode smoothness and stretch prompt ingestion. The chunk size is an SLO knob, and you should sweep it rather than accept a default.

Goodput is workload-specific and decays. A model update, a new prompt template that adds a 2,000-token system prompt, or a new customer with longer documents moves the curve. Rerun the sweep on every material change, and track a continuously measured production goodput proxy: the highest request rate seen in a window with attainment still above target.

SLO gaming. Optimizing the number instead of the experience happens quietly. Truncating outputs to improve TPOT, routing hard requests to a slower unmonitored path, or excluding timeouts all improve the dashboard while worsening the product. Pair latency SLOs with a quality check and with a count of truncated or refused requests.

Multi-tenant fairness. One tenant sending 30,000-token prompts can destroy another’s TTFT in a shared pool. Per-tenant token-rate limits and priority scheduling are part of the SLO design, not an afterthought.

Over-fitting to p95. If you tune only the p95, the p99 and the timeouts will betray you. Look at the tail distribution at least once per release, even if the SLO is written at p95.

Practical Recommendations

Start by writing the SLO down per surface with explicit definitions of where TTFT and TPOT are measured, what counts as a miss, and which attainment fraction applies. Make the attainment joint and per request. Resist a single global target.

Then measure rather than compute: run an open-loop sweep with production-shaped length distributions, find the knee, and call that the replica’s goodput. Plan to a fraction of it, add redundancy, and validate with a trace replay that includes losing a replica. Revisit after any model, template, engine-version or traffic-mix change.

Instrument at two layers. Scrape engine histograms for diagnosis, and emit a gateway-level per-request SLO counter for the number that goes in the budget. Alert on multi-window burn rates for paging and on queue depth for early warning, and write runbooks listing the cheap levers first.

A short checklist:

[ ] SLO classes defined per product surface, with TTFT, TPOT and E2E definitions
[ ] Joint attainment computed per request, failures and timeouts counted as misses
[ ] Open-loop load sweep with realistic length distributions, cold and warm cache
[ ] Goodput read at the knee, repeated three times, variance under 10 percent
[ ] Replica plan uses a utilization factor below 1 plus N+1 redundancy
[ ] Trace replay at 1.3x peak, including a replica-loss test
[ ] Multi-window burn-rate alerts plus a queue-depth early warning
[ ] Re-baseline procedure triggered by model, prompt-template or engine changes

Frequently Asked Questions

What is the difference between TTFT, TPOT and inter-token latency?

TTFT is the time from sending a request to receiving the first output token, covering queueing and prefill. TPOT is a per-request average of the time per generated token after the first, computed as (end-to-end latency minus TTFT) divided by (output tokens minus one). Inter-token latency (ITL) is the distribution of individual gaps between streamed chunks, pooled across requests. TPOT summarizes smoothness per request, while ITL reveals stalls that the average hides.

What is goodput in LLM serving and how is it different from throughput?

Throughput counts all tokens or requests served per second regardless of how long users waited. Goodput, as defined in the DistServe paper, is the maximum request rate that can be served while meeting the SLO attainment goal, such as 90 percent of requests, normalized per GPU. Throughput keeps rising while latency collapses, but goodput falls once too many requests miss their objectives, so goodput is the safer basis for capacity planning and cost per query.

What are good TTFT and TPOT targets for a chat application?

There is no universal standard. Illustrative starting points for chat are a p95 TTFT of roughly 0.8 to 1.5 seconds and a p95 TPOT of roughly 30 to 60 milliseconds, but you should calibrate them against your users and your model. Voice products need much tighter TTFT, while agent and batch workloads are better governed by end-to-end budgets. Set targets per product surface, and write down exactly where each metric is measured.

How many GPUs do I need for an LLM inference service?

Do not derive it from peak throughput. Load test one replica with an open-loop sweep and production-like prompt and output lengths, find the highest request rate where your joint SLO attainment stays above target, and call that goodput per replica. Divide forecast peak traffic by goodput times a utilization factor such as 0.7, then add N+1 redundancy. Finally, validate by replaying a production trace at above-peak load.

Which tools should I use to load test LLM inference?

Use vllm bench serve for quick checks against a vLLM server and GuideLLM for SLO-oriented evaluation with sweep, Poisson and constant-rate profiles and full TTFT, ITL and end-to-end distributions. Whatever the tool, use an open-loop arrival process, sample realistic length distributions, control for prefix cache state, and compute joint SLO attainment yourself from the per-request output so the definition is unambiguous.

How should I alert on LLM latency SLO violations?

Use multi-window, multi-burn-rate alerts on a per-request good/bad counter measured at the gateway, as described in the Google SRE Workbook, rather than alerting on raw p95 thresholds. Page on fast burn over short windows, ticket on slow burn over days, and add a queue-depth or waiting-request alert as an early signal. Segment by traffic class and prompt length so a failing slice is not averaged away.