LLM Batch Inference: Batch APIs, Throughput Scheduling and Cost Engineering

Most teams discover llm batch inference the expensive way. They write a loop that calls a chat endpoint two million times to classify, extract or summarise a backlog, hit rate limits at hour two, add retries, and finish a week later with a bill at full interactive prices and a results table nobody can fully reconcile. The work never needed an answer in two seconds. It needed an answer by tomorrow, at the lowest price per token, with every record accounted for.

This matters now because offline work has become the largest and least glamorous slice of LLM spend: evaluation sweeps, synthetic data generation, document enrichment, embeddings backfills, nightly classification and agent-trace grading. Every major provider sells a discounted asynchronous tier, and self-hosted engines have scheduling modes that behave very differently when no human is waiting.

You will leave with a decision framework for provider batch APIs versus self-hosted offline serving, the scheduling mechanics that make throughput-optimised serving cheap, an idempotent job design with checkpointing and reconciliation, and a worked, clearly labelled illustrative cost model with runnable code.

What this covers: why batch is a different problem from serving, current provider batch API terms, self-hosted throughput scheduling, a fault-tolerant pipeline, eval-on-batch, cost modelling, failure modes and a launch checklist.

Context and Background

Interactive serving and offline processing optimise opposite objectives. An interactive endpoint is judged by time to first token and per-token latency at a percentile; our post on LLM inference SLOs, TTFT, TPOT and goodput covers that regime. To meet those objectives a server must keep batches small enough that each decode step stays fast, and it must leave headroom for bursts. Both choices waste hardware.

Offline processing removes the latency objective entirely. The only questions are how many tokens per dollar you can push through and whether all the work finishes before a deadline measured in hours. That single change unlocks levers that interactive serving cannot use: running at the largest batch the memory allows, queueing work until capacity is cheap, and reordering requests to improve cache reuse.

Providers monetise that slack. As of the date of this post (10 November 2026), the vendor documentation I checked advertises roughly half price for asynchronous batch work: OpenAI’s Batch API and Anthropic’s Message Batches API state a 50 percent discount versus synchronous pricing, and Google’s Gemini API documents batch usage at half the interactive price for the same model. Amazon Bedrock markets batch inference for selected foundation models at around 50 percent of on-demand pricing, with the list of eligible models varying by region. Treat all of these as dated facts and re-check before budgeting, because discounts and model eligibility change.

A related lever is prompt reuse. Many offline jobs share a long instruction block and few-shot examples across millions of rows, which is exactly the situation where caching pays. Our analysis of prompt caching economics across providers covers the break-even math, and the existing deep dive on offline batch inference architecture, provider APIs versus self-hosted vLLM compares the build options. This post takes a different angle: the control plane. Discounts are the easy part; the engineering is sharding, idempotency, retry budgets, reconciliation and verifying quality at scale.

The thesis, which I will defend with numbers below, is that batch cost is a product of three factors you control separately: the unit price (provider tier or GPU hour), the token volume (prompt design and caching), and the waste factor (retries, duplicated work, failed records re-run from scratch). Most teams optimise the first, ignore the second, and never measure the third.

The Batch Inference Reference Architecture

Direct answer: LLM batch inference is asynchronous, deadline-driven processing of many independent requests where throughput and cost matter more than latency. A reference design has five parts: a request manifest, a sharder, an executor (provider batch API or self-hosted engine), a durable results store keyed by request ID, and a reconciler that re-queues anything missing or invalid.

Figure 1: Reference architecture for llm batch inference, with a durable manifest, shards, an executor and a reconciler closing the loop.

The diagram shows the data flow. A manifest lists every unit of work with a stable ID. A sharder cuts the manifest into jobs sized to the executor’s limits. The executor, either a provider batch endpoint or a self-hosted engine, processes shards asynchronously. Results land in a store keyed by the same ID, and the reconciler compares manifest to store, validates outputs and creates a new, smaller manifest for whatever is missing. The loop terminates when the missing set is empty or a retry budget is spent.

The manifest is the source of truth

Every request gets a deterministic identifier, usually a hash of the input content plus the prompt version plus the model name. Deterministic IDs give you three properties for free. Duplicate inputs collapse into one call. A re-run after a crash recognises completed work by lookup rather than by guessing. And two runs with different prompt versions never overwrite each other, which makes A/B comparison of prompts a join rather than a migration.

Keep the manifest in a database or object store, not in memory or in the batch job itself. Provider batches are scoped objects with their own retention windows; Anthropic documents that batch results are downloadable for 29 days from creation, measured from creation rather than completion. Your own store outlives that window and is the only place where “done” is defined by your validation rules rather than the provider’s status field.

Sharding to executor limits

Provider limits shape shard size. Per the documentation checked for this post, OpenAI’s Batch API accepts up to 50,000 requests and a 200 MB input file per batch, with a 24-hour completion window and the /v1/chat/completions and /v1/embeddings endpoints among those supported. Anthropic allows up to 100,000 Message requests or 256 MB per batch, whichever limit is reached first. Gemini’s file-based batch input is capped at 2 GB per file, with inline requests limited to 20 MB in total. Bedrock takes JSONL from S3 and records per-record errors in its output. Re-verify each limit at build time rather than hard-coding it.

Smaller shards cost you little and buy you a lot. A shard of 20,000 requests that fails validation or expires wastes 20,000 requests of retry work; a shard of 100,000 wastes five times as much. Shards also let you pipeline: submit the next while polling the last, and spread work across providers or accounts when per-batch enqueue limits bite. I default to shards of 10,000 to 25,000 requests, adjusted so that a typical shard finishes in well under the completion window.

Heterogeneous executors behind one interface

The executor is deliberately pluggable. A provider batch API is the right default when your model is a frontier hosted model, your volume is moderate, and operations headcount is scarce. A self-hosted engine such as vLLM, running an open-weight model on rented or spot GPUs, is right when volume is very high, the model is small enough to run efficiently, or data cannot leave your network. A single submit(shard) -> handle, poll(handle) -> status, fetch(handle) -> results interface lets one control plane drive both, and lets you route by cost: send what the hosted model must do to the hosted batch API, and bulk simple work to your own cluster. If you already route between models online, the same ideas appear in our write-up on LLM routing and model cascades.

Why the results store must be append-only and idempotent

Provider output ordering is explicitly not guaranteed. OpenAI states output line order may not match input order, and Anthropic says results must be matched by custom_id. The results store therefore upserts by request ID, and a second delivery of the same result is harmless. Write the raw provider payload alongside the parsed fields so you can re-parse later when your schema changes without paying for inference again. That single habit has saved more money in my experience than any discount tier.

Provider Batch APIs: Terms, Limits and What They Hide

Direct answer: Provider batch APIs trade latency for price. Typical terms in late 2026 are a 50 percent discount, a 24-hour target completion window, file or request-count limits per batch, and best-effort completion. Differences that matter in practice are partial-failure semantics, caching interaction, retention of results, and which endpoints are supported.

The table summarises what I verified from official documentation on 10 November 2026. Where a field was not stated in the pages I could read, I have written “not verified” rather than guess.

Provider	Discount	Window	Per-batch limits	Input format	Notable behaviour
OpenAI Batch API	50% vs sync	24 h, only option	50,000 requests, 200 MB	JSONL with custom_id, method, url, body	Separate rate pool from sync; one model per file; expired requests listed in error file
Anthropic Message Batches	50% of standard price	Expires at 24 h; most finish under 1 h	100,000 requests or 256 MB	Requests array with custom_id and params	Stacks with prompt caching; results kept 29 days; not ZDR or HIPAA eligible
Gemini API batch	Half interactive price	24 h target	2 GB file; 20 MB inline	Inline requests or JSONL via File API	Job expires after 48 h pending or running; context caching supported
Amazon Bedrock batch	About 50% of on-demand for selected models	Minutes to hours	Not verified	JSONL in S3 with recordId and modelInput	Per-record error field; PartiallyCompleted status; needs IAM service role
Vertex AI Gemini batch	Not verified	Not verified	Not verified	Cloud Storage or BigQuery	Docs describe BigQuery and Cloud Storage sources

Table 1: Provider batch terms checked against vendor documentation on 10 November 2026. Re-verify before relying on any figure.

Partial failure is the normal case

Each provider has a different notion of “the batch finished”. OpenAI’s batch moves through validating, in_progress, finalizing and then completed, expired or failed. When a batch expires, unfinished requests are cancelled, completed responses stay in the output file, and you are charged for completed work; the expired requests appear in the error file with a batch_expired code. Anthropic reports a per-request result type of succeeded, errored, canceled or expired, and states that errored, canceled and expired requests are not billed. Bedrock adds a PartiallyCompleted job state.

The consequence for design is simple: never treat a batch as one unit of success. Treat it as a bag of independent per-record outcomes, ingest whatever succeeded immediately, and compute the missing set. The reconciler in Figure 1 exists because the provider will hand you a mixed result and expect you to cope.

Caching and batch discounts stack, but not reliably

Anthropic’s documentation says the batch discount and prompt caching discounts can be combined, but that cache hits are best-effort because batch requests are processed concurrently in no guaranteed order, and quotes typical hit rates of 30 to 98 percent depending on traffic patterns. It recommends identical cache-control blocks in every request, and suggests the 1-hour cache duration because batches can take longer than the 5-minute default. Gemini supports referencing a cached-content resource in individual batch requests.

For cost modelling, this means you must not assume a 100 percent hit rate. Model the cache hit rate as a range and report the spread. Prompt layout still matters enormously: put the long shared instructions and examples first, and the per-row content last, so the shared prefix is byte-identical across requests. The mechanics of prefix reuse inside engines are covered in our post on prefix caching with RadixAttention and vLLM automatic prefix caching.

What batch APIs do not support

Batch endpoints lag the synchronous surface. Anthropic’s documentation lists streaming as unsupported in batches, along with the fast output mode, server-side fallbacks and a zero max_tokens cache pre-warm; it also notes a beta header that raises the output limit for batch requests only. OpenAI’s batch endpoints listed in the guide I read were chat completions and embeddings, with a single model per input file. If your pipeline depends on a feature that has just launched, check batch support before you design around it.

Data governance also differs. Anthropic states that batch data is stored for up to 29 days and that batches are not eligible for zero data retention or HIPAA arrangements. For regulated workloads that can be the deciding factor that pushes work to a self-hosted engine or to a cloud-provider-hosted route with different terms. Read the retention clause before you read the price.

Rate limits behave differently

Batch traffic generally draws on a separate pool from interactive traffic, which is one of its hidden benefits: a nightly job no longer starves your production chat endpoint. OpenAI documents a separate batch pool, with limits on enqueued prompt tokens per model and no cap on output tokens. Anthropic applies rate limits to both batch HTTP requests and the number of requests waiting to be processed, and warns that processing can slow in periods of high demand, increasing expirations. Plan for queue-depth limits when you pipeline shards: submitting twenty shards at once may simply fail the later ones.

Throughput-Optimised Serving: How Self-Hosted Offline Inference Works

Direct answer: Throughput-optimised serving maximises tokens per GPU-second by running the largest batch that fits in memory, keeping the GPU saturated with continuous batching, ordering work to reuse cached prefixes, and ignoring time to first token. The levers are batch size limits, scheduler token budgets, prefix sharing, length bucketing and parallelism choices.

Figure 2: A throughput-optimised scheduler for offline LLM inference, from length-sorted queue to continuous batching with prefix reuse.

Why large batches are cheap in decode

Decode is memory-bandwidth bound. Each step streams the model weights from high-bandwidth memory (HBM) once, whether it is generating one token or sixty-four. Adding sequences to the batch therefore raises the arithmetic done per byte of weights moved, and tokens per second grow almost linearly until one of two limits bites: the GPU becomes compute-bound, or the key-value (KV) cache runs out of memory. An interactive server stops growing the batch early because each additional sequence lengthens every step and hurts inter-token latency. An offline server has no such constraint and pushes until memory or compute is the wall.

In vLLM the main knobs are max_num_seqs, the cap on concurrent sequences, and max_num_batched_tokens, the per-step token budget that includes prefill chunks. Raising both and giving the engine more of the device memory through gpu_memory_utilization generally increases throughput, at the price of latency you do not care about. Engine defaults are tuned for general serving, so a throughput run usually benefits from deliberately larger values; measure on your model and hardware rather than copying someone else’s numbers.

Continuous batching and the tail problem

Static batching waits for the slowest sequence in a batch, which wastes slots when output lengths vary. Continuous batching, popularised by Orca and now standard in vLLM, SGLang and TensorRT-LLM, admits a new request the moment an old one finishes. For offline work this means the GPU stays full except at the very end of the run, when the queue drains and only a few long generations remain. That drain tail can consume a surprising fraction of wall-clock time, so cap max_tokens firmly and split the corpus so that the final shard does not hold the longest requests.

Sort by length, then shuffle within buckets

Sorting by prompt length before submission reduces padding waste in engines that pad, improves prefill batching, and makes memory use predictable. The risk is bias and correlated failure: if every long document lands in the same shard and that shard fails, retry cost spikes, and any drift over wall-clock time (a changed model revision, a throttled node) correlates with document length. A pragmatic approach is bucket by length into a handful of classes, then shuffle within each bucket, and interleave buckets across shards so each shard has a representative mix. Output length is usually unknown in advance; if you can predict it from the task (classification emits a few tokens, summarisation hundreds), bucket on predicted output length as well, because the decode tail is driven by output.

Offline corpora often share a long system prompt, a schema, or few-shot examples. Automatic prefix caching reuses the KV blocks of that shared prefix, so prefill compute for the shared part is paid once per cache residency rather than once per request. To exploit it, keep the shared block byte-identical and first, put per-row content last, and process requests that share prefixes close together in time so cached blocks are not evicted between uses. Radix-tree structures in SGLang and block hashing in vLLM both support this; our article on prefix caching explains the data structures.

A worked intuition, labelled illustrative: if a 2,000-token shared prefix is followed by 300 unique tokens, prefill cost per request falls from roughly 2,300 tokens of compute to roughly 300 when the prefix is resident. Because prefill is compute-bound and decode is bandwidth-bound, trimming prefill frees compute but does not speed decode; the saving shows up as higher requests per second only when prefill was a meaningful share of step time. For short-output tasks such as classification, it is the dominant share.

Chunked prefill, parallelism and engine choice

Chunked prefill splits long prompts into pieces that are interleaved with decode steps, smoothing step time. For offline work its value is lower than online, because you are not protecting a latency target, but it still helps keep memory use bounded; see our deep dive on chunked prefill and prefill-decode interference. For parallelism, prefer data parallelism (independent replicas) whenever the model fits on one GPU. Replicas share nothing, scale linearly, and a failed replica loses only its shard. Use tensor parallelism only when a model does not fit, because the all-reduce communication that it adds is pure overhead for a throughput job. For mixture-of-experts models the trade-off is more complex; see our comparison of tensor, expert and data parallelism for MoE serving.

Quantisation is another throughput lever. Lower-precision weights shrink the bytes moved per decode step and free memory for a larger KV cache, which raises the feasible batch size. The cost is a quality change you must measure on your own task, which brings us to eval-on-batch below.

The simplest self-hosted runner

vLLM ships an offline Python interface and a batch runner that consumes the same JSONL request format as the OpenAI Batch API, which makes the hosted-to-self-hosted switch cheap. The following sketch uses the offline LLM class with a shared prefix and length-bucketed prompts; verify argument names against the vLLM version you install, since engine arguments evolve.

from vllm import LLM, SamplingParams
import json, sys

SHARED_PREFIX = open("instructions.txt").read()   # identical bytes in every request

def build_prompt(row):
    # shared prefix first, per-row content last to maximise prefix cache hits
    return f"{SHARED_PREFIX}\n\n### Document\n{row['text']}\n\n### Answer\n"

llm = LLM(
    model="your-org/your-open-weight-model",
    enable_prefix_caching=True,
    max_num_seqs=256,              # tune upward until KV cache is the limit
    max_num_batched_tokens=16384,
    gpu_memory_utilization=0.92,
)
params = SamplingParams(temperature=0.0, max_tokens=256)

rows = [json.loads(l) for l in open(sys.argv[1])]
rows.sort(key=lambda r: len(r["text"]))            # length bucketing (then shard)
outputs = llm.generate([build_prompt(r) for r in rows], params)

with open(sys.argv[2], "w") as out:
    for r, o in zip(rows, outputs):
        out.write(json.dumps({"id": r["id"], "text": o.outputs[0].text,
                              "finish": o.outputs[0].finish_reason}) + "\n")

Note the finish field: a length finish reason means the generation was truncated at max_tokens, which should be a validation failure, not a success.

Spot and preemptible GPUs

Offline work is the ideal tenant for spot capacity. Interruptible instances are typically sold at a substantial discount to on-demand, and since no user is waiting, preemption costs only the work in flight. The condition is that the job is checkpointed at shard granularity: each worker pulls a shard, writes results atomically, and acknowledges completion. A worker that is preempted loses at most one shard, and a lease timeout returns the shard to the queue. The same principles, applied to training, are in our post on spot GPU checkpointing and preemption handling. Keep shards small enough that losing one costs minutes, not hours.

Building a Fault-Tolerant Batch Pipeline

Direct answer: A reliable batch pipeline is a state machine over request IDs, not a script. Each record moves through pending, submitted, succeeded, invalid or dead-lettered, transitions are written durably, and a reconciler loop re-queues only what is missing, within a bounded retry budget.

Figure 3: Per-record state machine and retry flow for an idempotent LLM batch pipeline.

The state machine

Model every record with five states. Pending means the record is in the manifest and not yet in any shard. Submitted means it is inside a live provider batch or worker lease. Succeeded means a valid result is stored. Invalid means a result arrived but failed validation. Dead-lettered means the retry budget is spent and a human or a different strategy must intervene. Only the reconciler moves records between states, and every move is an upsert keyed by request ID, so crashes and duplicate deliveries do not corrupt anything.

Retry budgets and error classification

Not all failures deserve a retry. Classify them. Transient failures, such as an expired batch, an overloaded provider or a preempted worker, are retried with backoff and the same payload. Deterministic input failures, such as a prompt that exceeds the context window or content that triggers a policy refusal, will fail identically every time; retrying burns money. Truncated or malformed outputs are model-behaviour failures; retry once with a modified setting (larger max_tokens, a stricter format instruction, temperature zero) and then dead-letter.

A budget of two or three attempts per record and a global cap on retry spend, say ten percent of the first-pass cost, keeps a bad prompt from turning a hundred-dollar job into a thousand-dollar one. The cap is the circuit breaker: if retries exceed it, stop and inspect rather than loop.

Idempotency in practice

Idempotency means running the pipeline twice yields the same final state and charges for each unit of work once. The recipe is a deterministic request ID, an upsert-only results store, and a “skip if succeeded” check before every submission. With provider batches, the custom_id you set is your handle; keep it unique per batch as the docs require and encode the content hash and prompt version in it. If your process crashes between submitting a batch and recording the returned batch ID, you can orphan a paid job. Guard that window by writing an intent record with a client-generated shard ID before submission, and on restart list recent provider batches and match them to intents before submitting anything new.

Reconciliation as a first-class step

After each shard, compute four numbers: records expected, results received, results valid, and results missing. They must satisfy an invariant, expected equals valid plus invalid plus missing plus dead-lettered, and a violation is a bug in your pipeline, not in the provider. Assert it in code and fail loudly. Provider-side counters help, for example per-batch request counts for processing, succeeded, errored, canceled and expired, but your store is authoritative.

Reconciliation also catches silent problems that status fields hide: duplicate IDs in output, results whose content does not match the request (mixed-up ordering if you matched by line number instead of ID), and outputs that parse but are semantically empty. A cheap rule such as “classification label must be in the allowed set and confidence field present” catches a large share of them.

A runnable skeleton for a hosted batch API

The following Python sketch drives Anthropic’s Message Batches API with sharding, idempotent IDs and reconciliation. The same shape works for OpenAI or Gemini with different client calls. It is a teaching skeleton: add persistence, backoff and logging before production use, and verify SDK method names against the version you install.

import hashlib, json, time
import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-4-5"        # placeholder: substitute the model you are licensed for
PROMPT_VERSION = "v3"
SHARD_SIZE = 10_000

SYSTEM = [{"type": "text", "text": open("instructions.txt").read(),
           "cache_control": {"type": "ephemeral", "ttl": "1h"}}]

def request_id(row):
    h = hashlib.sha256(f"{MODEL}|{PROMPT_VERSION}|{row['text']}".encode()).hexdigest()
    return f"{PROMPT_VERSION}-{h[:40]}"      # deterministic, unique, short

def shards(rows, size):
    for i in range(0, len(rows), size):
        yield rows[i:i + size]

def submit(shard):
    reqs = [{"custom_id": request_id(r),
             "params": {"model": MODEL, "max_tokens": 256, "system": SYSTEM,
                        "messages": [{"role": "user", "content": r["text"]}]}}
            for r in shard]
    return client.messages.batches.create(requests=reqs)

def collect(batch_id, store):
    for entry in client.messages.batches.results(batch_id):
        rid, res = entry.custom_id, entry.result
        if res.type == "succeeded":
            store[rid] = {"status": "ok", "raw": res.message.model_dump()}
        else:                                    # errored, canceled, expired
            store.setdefault(rid, {"status": res.type})

def run(rows, store):
    todo = [r for r in rows if store.get(request_id(r), {}).get("status") != "ok"]
    for shard in shards(todo, SHARD_SIZE):
        b = submit(shard)
        while (b := client.messages.batches.retrieve(b.id)).processing_status != "ended":
            time.sleep(60)
        collect(b.id, store)
    missing = [r for r in rows if store.get(request_id(r), {}).get("status") != "ok"]
    return missing                                # feed back into run() under a retry budget

The loop is deliberately boring. It filters to the not-yet-successful set at the start, so re-running after a crash only pays for missing work, and it returns the missing set for the caller to retry under a budget.

Eval-on-Batch: Proving Quality at Scale Without Paying Twice

Direct answer: Evaluate batch outputs by sampling, not by rereading everything. Draw a stratified sample, score it with a rubric or a judge model, track agreement with a small human-labelled golden set, and gate the full run on those scores. Batch itself is also the cheapest way to run the evaluation, since eval sweeps are asynchronous too.

Batch runs fail quietly. A bad prompt does not throw an exception; it produces two million plausible, subtly wrong rows. Quality control therefore has to be a stage in the pipeline, with the same rigour as the reconciler. I use three layers.

Layer one: structural validation on every record

Every output passes deterministic checks: valid JSON against a schema, required fields present, enumerated labels in range, finish reason not length, length within bounds. These checks are essentially free and catch truncation, refusals and format drift. A rising invalid rate across shards is the earliest warning that something changed, such as a provider model revision or a corrupted slice of input.

Layer two: golden set and canary shard

Before launching the full corpus, run a canary shard of a few hundred records that includes a golden set with trusted labels. Compute accuracy or agreement and compare to a threshold set in advance. Only a passing canary unlocks the remaining shards. Repeat the golden set inside every later shard, perhaps one percent of records, so you get a quality time series and can detect drift mid-run. The cost is small and the information is large: you learn whether shard 40 is as good as shard 1.

Layer three: judge sampling with known biases

For open-ended outputs, sample a few hundred to a few thousand records stratified by input type and length, and score them with a rubric using a judge model, ideally also through the batch tier. Judges have known biases such as favouring longer answers and their own family’s style, so calibrate against human labels on a subset and report agreement alongside the score; our post on LLM-as-a-judge bias and calibration goes deeper, and the harness-level view is in evaluation harness architecture.

Sample size arithmetic

How many samples give a trustworthy accuracy estimate? For a binary correct-or-not metric, the standard error of a proportion near 0.9 with n samples is sqrt(0.9 x 0.1 / n). With n = 400 that is about 1.5 percentage points, so a 95 percent interval is roughly plus or minus 3 points. To detect a regression from 92 to 90 percent reliably you need more samples or a paired comparison on the same inputs. These are standard statistics; the point is to decide the sample size before the run, not after you see a disappointing number.

Comparing two models or prompts on the same corpus

Because batch is cheap, run both candidates on the identical sample and compare pairwise. Paired comparison removes input-difficulty variance and needs far fewer samples than two independent estimates. Then project cost and quality together: a smaller model that is two points worse and four times cheaper may be the right choice for a bulk enrichment task, and the wrong one for the extraction that feeds a regulated report. Cascading, where a cheap model handles confident cases and a stronger model the rest, is the logical extension and ties into the routing and cascade discussion.

Figure 4: Quality gates for LLM batch inference: structural validation, canary shard, golden-set drift tracking and sampled judging.

A Worked, Illustrative Cost Model

Direct answer: Batch cost equals tokens times unit price times a waste factor. A 50 percent batch discount, prompt caching on the shared prefix and a retry budget interact multiplicatively. In the example below, a synthetic two-million-document job falls from about 14,700 dollars at interactive prices to roughly 5,000 to 5,700 dollars on a batch API with caching, with output tokens dominating the remainder.

Every number in this section is illustrative. The prices are round placeholders chosen to make arithmetic checkable, not any vendor’s current list price. Substitute your own.

Assumptions

2,000,000 documents.
Each request has 900 shared prefix tokens (instructions and examples) plus 300 unique input tokens, and produces 250 output tokens.
Illustrative interactive price: 3 dollars per million input tokens and 15 dollars per million output tokens.
Illustrative batch tier: 50 percent of both.
Illustrative cache-read price: 10 percent of the base input price. Cache write premiums are ignored for simplicity, which slightly flatters the cached scenarios.

Baseline: interactive, no caching

Input tokens: 2,000,000 x 1,200 = 2.4 billion tokens, or 2,400 million. At 3 dollars that is 7,200 dollars. Output tokens: 2,000,000 x 250 = 500 million at 15 dollars is 7,500 dollars. Total: 14,700 dollars.

Batch tier only

Half price on everything: 7,350 dollars. This is the no-effort win: change the endpoint, accept up to a day of latency.

Batch plus prefix caching

Suppose the cache hits on a fraction h of the 900 shared tokens. At h = 1.0, per-document input cost is (900 x 0.3 + 300 x 3.0) / 1,000,000 = 0.00117 dollars interactive-equivalent, so 2,340 dollars for the corpus, halved by batch to 1,170 dollars. Output at batch price is 3,750 dollars. Total: about 4,920 dollars.

At h = 0.7, 630 tokens are read from cache and 570 are charged at full input price: (630 x 0.3 + 570 x 3.0) / 1,000,000 = 0.001899 dollars per document, so 3,798 dollars, halved to 1,899. Plus 3,750 for output: about 5,650 dollars.

At h = 0.3, 270 cached and 930 full: (270 x 0.3 + 930 x 3.0) / 1,000,000 = 0.002871 per document, 5,742 dollars, halved to 2,871; plus 3,750 gives about 6,620 dollars. The cache hit rate matters, but notice the floor: output tokens at batch price are 3,750 dollars no matter what, so caching can only attack the input share of spend.

Add the waste factor

If 4 percent of records need a retry and each retry costs the same as the original, the waste factor is 1.04. If you also discover after the run that the prompt was wrong and re-run everything, the factor is 2.04, which erases the entire batch discount. This is why canary shards and golden sets belong in the cost model: a 500-record canary costs a fraction of a cent per record at batch prices and prevents a full re-run.

Scenario (illustrative)	Input cost	Output cost	Total
Interactive, no cache	7,200	7,500	14,700
Batch only	3,600	3,750	7,350
Batch, cache hit 100% of prefix	1,170	3,750	4,920
Batch, cache hit 70%	1,899	3,750	5,649
Batch, cache hit 30%	2,871	3,750	6,621

Table 2: Illustrative cost of a two-million-document job under different tiers and cache hit rates, in dollars. Prices are placeholders, not vendor quotes.

The lever that dominates: output tokens

In this example output is 62 to 76 percent of the batch-with-caching bill. Prompt engineering for short outputs, structured output with terse field names, and tasks reframed as classification all attack the larger term. Cutting average output from 250 to 150 tokens saves 1,500 dollars at batch prices, more than going from a 30 to a 100 percent cache hit rate saves on input. Also pick a firm max_tokens; a handful of runaway generations are paid at the output rate.

Self-hosted break-even

For a self-hosted open-weight model the unit price is GPU cost divided by sustained tokens. Cost per million output tokens equals GPU dollars per hour divided by (tokens per second x 3,600 / 1,000,000), divided again by utilisation. Illustratively, a 2.50-dollar-per-hour GPU sustaining 2,500 aggregate output tokens per second at 70 percent utilisation yields 2,500 x 3,600 x 0.7 = 6.3 million tokens per hour, so about 0.40 dollars per million output tokens. A spot price at, say, 40 percent of on-demand would bring that near 0.16 dollars. These are not benchmarks; real throughput depends on model size, sequence lengths, quantisation, and engine tuning, and you must measure it.

The comparison is not apples to apples, because the self-hosted model is a different and probably less capable model than a hosted frontier model, and you inherit operations, evaluation and on-call costs. Self-hosting wins when volume is large and steady, the task is within a smaller model’s competence, or data residency forbids hosted APIs. Hosted batch wins when the task needs frontier capability, volume is bursty, and your team’s time is the scarcer resource. For an end-to-end view of per-token unit economics and gross margin, see our analysis of LLM token economics and unit economics of AI features.

Governance: budgets and kill switches

Batches may overshoot spend limits slightly, because they are accepted before cost is known. Anthropic notes batches can slightly exceed workspace spend limits, so enforce your own budget in the control plane: estimate cost per shard from token counts before submission, track cumulative spend, and refuse to submit when the projection exceeds the approved budget. If you already run a gateway for online traffic, route batch spend through the same accounting; our comparison of LLM gateways for routing, fallback and budgets is relevant.

Trade-offs, Gotchas, and What Goes Wrong

Latency is genuinely unbounded. A 24-hour window is a ceiling for the provider, not a promise of speed. Anthropic says most batches finish within an hour, but processing can slow under demand and cause more expirations. Never put a batch on the critical path of a user-facing flow, and give the job a deadline with slack, for example finish by 06:00 for a 09:00 report, so one slow shard does not break the business.

Expiry wastes calendar time, not money. Expired requests are generally not billed, which is kind to your budget, but you lose a day. Shard so that you can resubmit only the expired slice, and consider submitting early in the day so there is time for a second attempt.

Prompt changes invalidate caches and comparisons. Changing one byte of the shared prefix breaks cache hits for every request; changing prompts mid-run makes shards incomparable. Freeze the prompt version for a run and bake it into the request ID.

Ordering assumptions are bugs. Matching results to inputs by position works in tests and fails in production. Always match on the ID.

Hidden retention and compliance gaps. Batch data may sit in provider storage longer than synchronous traffic does, and some regimes such as zero data retention or HIPAA arrangements may not cover batch. Check legal terms before moving sensitive workloads.

Feature lag. New model features, streaming-only behaviours and some tool integrations may not be available in batch at launch. Test the exact request shape you plan to send on a 100-record shard.

The drain tail on self-hosted runs. The last 5 percent of requests often take 20 percent of wall-clock time because few sequences remain. Over-sharding helps; so does prioritising long generations first.

Spot interruptions without shard-level checkpoints. A job that holds results in memory until the end will lose everything to a single preemption. Persist per shard.

Silent quality drift. Provider model aliases can move to new revisions, and open-weight serving stacks change numerics between versions. Pin model versions where the provider allows it, and keep the golden set in every shard.

Anti-pattern: batch for everything. If the output feeds a user in seconds, or an agent loop needs the result before its next step, batch is the wrong tool. Likewise, tiny jobs of a few hundred requests gain little because engineering effort exceeds savings.

Practical Recommendations

Begin with the simplest executor that meets the deadline: a provider batch API for most teams. Stand up the control plane first, because it is reusable across providers and engines: manifest, deterministic IDs, shards, results store, reconciler. Then optimise tokens before optimising price. Shorten outputs, structure prompts so the shared prefix is identical and first, and use the longer cache duration where the provider offers one. Only then consider moving bulk, simple tasks to a self-hosted engine on spot GPUs, and only if measured volume justifies the operational load.

Treat quality as a pipeline stage rather than a post-mortem. Run the canary and golden set before the full corpus, repeat the golden records in every shard, and set pass thresholds in advance. Budget the retry spend explicitly, and report cost per valid result rather than cost per request, since a cheap run with a high invalid rate is not cheap.

Freeze model version, prompt version and parameters per run; encode them in the request ID.
Shard to 10,000-25,000 requests, below provider limits, and interleave length buckets.
Put shared instructions first and byte-identical; per-row content last.
Validate every record structurally; reject length finish reasons.
Run a canary shard with a golden set; gate the full run on it.
Cap retries at two or three per record and at roughly ten percent of first-pass spend.
Assert the reconciliation invariant after every shard.
Enforce your own spend cap; do not rely on provider limits.
Store raw responses so schema changes never require paying for inference twice.
Re-verify provider discounts, limits and retention terms on the day you budget.

Frequently Asked Questions

How much does LLM batch inference save compared with real-time calls?

Provider documentation checked on 10 November 2026 advertises about 50 percent off for OpenAI Batch, Anthropic Message Batches and Gemini batch, with Bedrock around 50 percent for selected models. Savings compound with prompt caching, which Anthropic says stacks with the batch discount, though cache hits are best-effort. Real savings also depend on your waste factor: retries and full re-runs can erase the discount, so measure cost per valid result.

How long does a batch API job take?

The ceiling is typically 24 hours, and OpenAI lists 24 hours as its only completion window. Anthropic states most batches finish in under an hour, but results arrive when everything completes or at 24 hours, and slowdowns under demand can cause expirations. Gemini documents a 24-hour target and expires jobs after 48 hours. Design for the worst case, with slack before any business deadline.

Should I self-host vLLM or use a provider batch API?

Use a provider batch API when you need a frontier hosted model, volume is bursty, or operations capacity is limited. Self-host when volume is large and steady, a smaller open-weight model meets quality needs, or data cannot leave your network. Compare cost per valid result at measured throughput, including evaluation and on-call effort, not just GPU hours against token prices. A common hybrid is hosted batch for hard cases and self-hosted for bulk work.

What happens to requests that fail inside a batch?

Failures are per record. OpenAI writes failed requests to an error file and lists expired ones with a batch_expired code; Anthropic marks each result succeeded, errored, canceled or expired and does not bill the last three; Bedrock can end in a PartiallyCompleted state with per-record error fields. Ingest successes immediately, compute the missing set by ID, classify failures, and retry only transient ones within a budget.

Does prompt caching work with batch requests?

On Anthropic, yes: the documentation says batch and caching discounts stack, but because requests are processed concurrently and in no guaranteed order, hits are best-effort, with typical rates of 30 to 98 percent. Use identical cache-control blocks, put shared content first, and consider the 1-hour cache lifetime. Gemini supports referencing cached content in batch requests. Always model hit rate as a range rather than assuming certainty.

How do I check quality across millions of batch outputs?

Layer the checks. Validate structure on every record, run a canary shard with a golden set before the full job, repeat golden records in every shard to detect drift, and sample stratified outputs for rubric or judge scoring calibrated against human labels. Decide sample sizes in advance: about 400 samples gives roughly plus or minus three points on an accuracy near 90 percent. Gate the full run on pre-set thresholds.