Offline Batch LLM Inference Architecture: Provider Batch APIs vs Self-Hosted vLLM (2026)

Most teams meet batch LLM inference the same way: a one-off job to label two million support tickets is run through the chat endpoint, takes nine days of babysitting against rate limits, and costs twice what it needed to. The workload never needed an answer in two seconds. It needed two million answers by Friday, correct, deduplicated, and cheap. Treating that as an interactive problem is the most common and most expensive mistake in applied LLM engineering.

In 2026 the choice has crystallised into two serious options. The three big API providers sell asynchronous batch endpoints at roughly half the interactive price, and open-source engines such as vLLM let you run the same job on GPUs you rent by the hour. Neither wins universally. The winner depends on volume, deadline tolerance, prompt shape and how well you can keep a GPU busy.

What this covers: when offline beats online, what the provider batch APIs actually guarantee (verified limits), the prefill and decode math behind self-hosted throughput, a worked break-even calculation, job-queue design with checkpoint and resume, prefix caching, spot GPUs, structured output, validation, and the failure modes that wreck long runs, with runnable Python for a sharded runner and a cost calculator.

Context and Background

Interactive serving optimises for latency under uncertainty. Requests arrive at random, each user is waiting, and the system keeps headroom so that a burst does not push time-to-first-token past a few hundred milliseconds. That headroom is paid for in idle silicon. Offline work inverts the objective: the whole dataset is known in advance, nobody is waiting on any single row, and the only metric that matters is cost per completed item subject to a deadline measured in hours.

That inversion is why a whole class of workloads belongs offline: evaluation suites run on every model or prompt change, enrichment of catalogue or CRM records, bulk classification and tagging, embedding backfills, document extraction over archives, and synthetic data generation for fine-tuning or distillation. They share three traits. The inputs exist before the job starts, the outputs are consumed by a pipeline rather than a person, and a late answer is only slightly worse than an early one.

The industry has converged on a standard answer for the first tier. OpenAI’s Batch API, Anthropic’s Message Batches API and Google’s Gemini Batch API all price asynchronous work at 50% of the interactive rate, in exchange for a completion window measured in hours (the specifics are in the next section). The second tier is DIY: an inference engine you control, fed from a queue you control. vLLM is the most widely deployed open-source option, and its offline LLM class is designed exactly for this: hand it a list of prompts and it schedules them with continuous batching until the list is drained. The mechanics of that scheduler are covered in our continuous batching architecture deep-dive, and engine selection is compared in vLLM vs SGLang vs TensorRT-LLM.

The thesis of this post is narrower than “batch is cheaper.” It is that batch inference is a data-engineering problem wearing an inference costume. Whichever backend you pick, the durable parts are the sharding scheme, the idempotency of items, the checkpoint format and the validation gate. The model call is the replaceable part. Teams that build that skeleton once can route the same job to a provider API on Monday and a spot-GPU fleet on Thursday, choosing by arithmetic rather than by habit. Cost levers that sit outside batching, such as model right-sizing and caching, are covered in AI inference cost optimization.

The Batch Inference Reference Architecture

Direct answer: A production batch LLM inference system has five parts: a sharder that gives every item a stable ID, a router that picks the backend by deadline and volume, an execution layer (provider batch API or self-hosted vLLM), an append-only result store keyed by item ID, and a validation gate that feeds retries and a dead-letter store. Idempotent items plus checkpointed shards make any backend swappable and any failure resumable.

Figure 1: Reference architecture for batch LLM inference. A stable-ID sharder feeds a router; three execution paths converge on one result store and one validation gate.

Figure 1 shows the flow. Every dataset row is assigned an item ID derived from content or source key, never from its position in a file. The sharder groups items into fixed-size shards. The router then decides where a shard runs, and all backends write into the same result store keyed by item ID, so downstream code never learns which backend produced a row. The long-description version: rows enter, are sharded and routed, executed by one of three backends, stored, validated, and either promoted to the downstream table or retried and eventually dead-lettered (full-size diagram).

Why stable item IDs are the foundation

Everything else in this architecture depends on one property: running the same item twice must be safe. With a stable ID and a result store that treats the ID as a primary key, a retry overwrites an identical record, a resume skips completed work, and a duplicate shard assignment costs money but never corrupts data. Position-based IDs break the moment you re-shard or re-sort. A content hash of the normalised input plus the prompt template version works well, because it also gives you free deduplication: if two rows hash identically, you pay for one.

Include the prompt version and model identifier in the key space, not just the input hash. When you change the prompt, you want old results to be recognisably stale rather than silently reused, and you want the ability to run two prompt versions side by side in an evaluation. A composite key such as sha256(template_version | model | normalised_input) costs nothing and prevents an entire category of “which version produced this row?” incidents.

The router: deadline, volume and prompt shape

The router is a few lines of policy, but the policy encodes the economics. Small or urgent shards go to the online API at full price, because the fixed overhead of a batch job (file upload, validation, polling) outweighs a 50% discount on a few thousand tokens. Bulk work that tolerates a day goes to a provider batch API. Very large and steady workloads, especially ones with a long shared prefix or a model the providers do not host, go to self-hosted engines. A fourth consideration is data governance: if rows may not leave your network, the router has one legal answer regardless of price.

The router should also be able to fall back. A provider batch that expires unfinished returns its completed items and cancels the rest, so the leftover shard can be re-routed to the online API when a deadline is close, paying full price only for the tail. We return to this “tail rescue” pattern in the failure-modes section.

Provider batch APIs: what is actually guaranteed

The three major providers are more alike than different, and the differences matter at the edges. The figures below come from each provider’s documentation as checked in November 2026; limits change, so verify before you size a job.

OpenAI Batch API. Batches are billed at 50% below the synchronous price and complete within a 24-hour window, which the documentation states is the only window currently available, adding that jobs often finish sooner. A single batch accepts up to 50,000 requests with an input file up to 200 MB, and each file targets a single model. Supported endpoints include /v1/responses, /v1/chat/completions, /v1/embeddings, /v1/completions and /v1/moderations, plus image generation and edit endpoints. Batch usage does not consume your normal per-model rate limits, but each model has a cap on enqueued prompt tokens visible in your account settings, and you can create up to 2,000 batches per hour. Output files are deleted 30 days after completion. If a batch expires, unfinished requests are cancelled, you are charged for completed ones, and the rest appear in the error file with the code batch_expired.

Anthropic Message Batches. All usage is charged at 50% of standard prices. A batch holds up to 100,000 requests or 256 MB, whichever is hit first. Batches expire if processing does not finish within 24 hours, though the documentation says most finish in under an hour. Results are downloadable for 29 days counted from creation, not from completion, which matters if you queue a large batch and forget it. Each result carries a custom_id and a type of succeeded, errored, canceled or expired; only succeeded requests are billed, and results arrive in arbitrary order so matching on custom_id is mandatory. Prompt caching works inside batches on a best-effort basis; the documentation cites typical hit rates between 30% and 98% and recommends the one-hour cache duration for batches that share context, because processing can outlast the default five-minute lifetime. The discounts stack with caching discounts.

Google Gemini Batch API. Batch jobs cost 50% of the interactive price for the same model, with a stated target turnaround of 24 hours. Inline requests must fit in 20 MB total; larger jobs use a JSONL input file of up to 2 GB uploaded through the File API. A job that remains pending or running beyond 48 hours moves to JOB_STATE_EXPIRED with no retrievable results, which is a harsher failure than the other two providers’ partial-result behaviour, so size Gemini jobs conservatively and shard them. Vertex AI offers an equivalent batch prediction surface for Gemini under Google Cloud’s own quota and billing; check its current pricing page rather than assuming parity with the developer API.

Three architectural consequences follow. First, provider batches are not a streaming interface: you cannot start consuming results until the batch ends (or, for some, partially before), so downstream stages must tolerate burst arrival. Second, the 24-hour window is a ceiling, not a typical latency, but you must design for the ceiling. Third, request-count caps mean that a ten-million-row job is at minimum a hundred or two hundred batches, which makes your own sharding and tracking layer mandatory even in the “simple” managed path.

Throughput Math: Prefill, Decode and Break-Even Utilisation

Self-hosting only wins if you keep the GPU busy, so you need a model of what “busy” yields. The model starts with the two phases of transformer inference, and their different bottlenecks explain nearly every tuning decision in batch inference.

Figure 2: Prefill is compute-bound; decode is memory-bound until batching amortises the weight reads, after which throughput plateaus at a compute or KV-cache limit.

During prefill, all prompt tokens are processed in parallel through large matrix multiplications. The arithmetic intensity is high, so prefill saturates the GPU’s compute units and its throughput in tokens per second is relatively insensitive to batch size once a few thousand tokens are in flight. During decode, each sequence produces one token per step, and each step must read the full set of model weights from high-bandwidth memory. For a single sequence this is a memory-bandwidth problem: the GPU spends most of its time moving weights, not computing. Batching N sequences into one step reads the weights once and produces N tokens, so decode throughput climbs nearly linearly with batch size, until one of two ceilings is hit.

The first ceiling is compute: at some batch size the matrix multiplications themselves become the bottleneck and extra sequences add proportional time. The second, usually reached first for long contexts, is KV-cache capacity. Every in-flight sequence holds key and value tensors for every token it has seen. The cache size per token is roughly 2 x layers x kv_heads x head_dim x bytes_per_element, so a long-context job fills memory with a handful of sequences. Offline jobs push against this limit hard because they want the batch as large as memory allows, which is why vLLM’s paged KV allocation and its gpu_memory_utilization setting dominate tuning (full-size diagram).

Why offline engines beat online engines on the same GPU

The offline setting removes the constraints that make online serving inefficient. There is no per-request latency SLO, so the scheduler can keep the maximum number of sequences resident and run the engine at its throughput optimum rather than its latency optimum. There is no arrival process, so queue depth never drops to zero until the very end. And you control the order: sorting or bucketing prompts by length reduces padding waste and makes the memory footprint predictable, and grouping by shared prefix raises cache hit rate. It is common for the same hardware to deliver a large multiple of its online per-GPU throughput when driven offline, although the multiple depends on model, context length and hardware, and I will not quote a universal figure. Measure your own workload.

A worked break-even (illustrative numbers)

All figures in this subsection are illustrative assumptions, chosen to show the method; substitute your own measurements. Suppose a mid-sized open model on a single rented GPU at \$2.50 per hour, with measured offline engine throughput of 20,000 prompt tokens per second in prefill and 2,500 generated tokens per second in decode at the chosen batch size. The job has items of 1,000 input tokens and 200 output tokens each.

Per item, prefill takes 1,000 / 20,000 = 0.05 seconds and decode takes 200 / 2,500 = 0.08 seconds, for 0.13 seconds of GPU time. One GPU-hour therefore completes about 3,600 / 0.13 = 27,700 items. At 100% utilisation the cost is \$2.50 / 27,700, or about \$90 per million items.

Now price the same job on a provider batch API with an assumed post-discount rate of \$0.15 per million input tokens and \$0.60 per million output tokens. One item costs 1,000 x 0.15 / 1,000,000 + 200 x 0.60 / 1,000,000 = \$0.00015 + \$0.00012 = \$0.00027, so a million items cost \$270. The break-even GPU utilisation is the ratio of the two: \$90 / \$270, or about 33%. Below one-third effective utilisation, the API is cheaper; above it, self-hosting wins, and at 80% the self-hosted cost is about \$113 per million items, a 58% saving.

“Effective utilisation” is the number people overestimate. It is not the nvidia-smi utilisation figure, which counts any kernel running. It is the fraction of paid GPU-hours spent producing useful output at the measured peak rate. Model loading, which can take minutes for large checkpoints, idle time waiting for the queue, tail-end under-filled batches, retries of failed items and preemption losses all subtract from it. A job that runs for six hours but spends 40 minutes loading and 50 minutes in a half-empty tail is at 75% before any engineering inefficiency. Add engineer time, and the honest comparison for small jobs favours the API almost every time.

Job Queue Design: Sharding, Idempotency, Checkpoints and Dead Letters

The queue is where batch projects succeed or fail, because a ten-hour job on preemptible hardware will be interrupted at least once. The design goal is that no interruption costs more than one shard of rework, and that the system never needs a human to decide what was done (full-size diagram).

Figure 3: Shard-lease worker loop. Workers skip items already in the done log, results append durably, and expired leases return shards to the queue.

Shard size is a trade-off, not a constant

A shard is the unit of leasing, checkpointing and retry. Small shards (hundreds of items) mean cheap recovery and fine-grained progress, but more scheduling overhead and, for provider batches, more files and polling. Large shards (hundreds of thousands) amortise overhead but make a single failure expensive and starve the tail of parallelism. A workable rule is to size a shard to roughly five to fifteen minutes of work on one worker, so lease durations, heartbeats and losses stay human-scale. For provider batch APIs the shard is bounded by the request caps, but you will usually choose far smaller shards than the cap to bound the blast radius of an expired batch.

Idempotent items, append-only results, and a done log

Each worker writes results as JSONL records containing the item ID, the output, token counts, model and prompt version, and a status. The shard’s done log is simply the set of item IDs with terminal status in that shard’s result file. On resume, the worker reads the log, subtracts completed IDs, and processes only the remainder. Because writes are append-only and keyed by ID, a crash mid-write leaves at worst a truncated final line, which the reader must tolerate by ignoring any line that fails to parse. Do not rewrite result files in place; do not hold results in memory until the end of a shard.

Flush on a count-or-time trigger, for example every 200 items or 30 seconds, and fsync or the object-store equivalent. That interval is the maximum rework after a hard kill. Object stores without append semantics need either one small object per flush or a local append file synced to storage periodically; either works as long as the reader treats the union of all parts as the log.

Retries, poison items and the dead-letter store

Failures fall into three kinds and deserve different treatment. Transient failures (a 429, a 5xx, a dropped connection, an engine restart) are retried with exponential backoff and jitter. Item-specific failures (a prompt that exceeds the context window, a malformed document that crashes the tokenizer, an output that never validates) are poison items: retrying them forever blocks the shard. Systemic failures (bad credentials, a wrong model name) should stop the whole job quickly rather than burn the retry budget on every item.

Give each item an attempt counter and a small retry budget, commonly three. When the budget is spent, the item and its last error go to a dead-letter store and the shard completes anyway. This keeps one pathological row from holding two hundred thousand healthy ones hostage. Review dead letters in aggregate: a cluster that shares a trait, such as all documents above a length or in one language, is usually a prompt or preprocessing bug, and fixing it and replaying the dead-letter set is cheap.

A runnable sharded runner with checkpoint and resume

The following Python is a self-contained skeleton. It uses only the standard library, writes per-shard result and dead-letter files, leases shards through lock files with expiry, and takes the backend as a plain function so you can plug in a provider SDK or a vLLM LLM.generate call. It has been written for clarity, not tuned for scale; on object storage you would replace the lock-file lease with a conditional write or a queue visibility timeout.

import hashlib, json, os, time, random
from pathlib import Path

def item_id(template_version: str, model: str, text: str) -> str:
    norm = " ".join(text.split())
    return hashlib.sha256(f"{template_version}|{model}|{norm}".encode()).hexdigest()[:20]

def make_shards(rows, out_dir: Path, shard_size=500, template_version="v1", model="m"):
    """rows: iterable of dicts with a 'text' field. Writes shard_XXXX.jsonl, deduplicating by item id."""
    out_dir.mkdir(parents=True, exist_ok=True)
    seen, shard, n = set(), [], 0
    def flush():
        nonlocal shard, n
        if shard:
            with open(out_dir / f"shard_{n:05d}.jsonl", "w") as f:
                for r in shard: f.write(json.dumps(r) + "\n")
            shard, n = [], n + 1
    for r in rows:
        iid = item_id(template_version, model, r["text"])
        if iid in seen: continue
        seen.add(iid); shard.append({"id": iid, "text": r["text"]})
        if len(shard) >= shard_size: flush()
    flush()
    return n

def read_jsonl(path: Path):
    if not path.exists(): return []
    out = []
    with open(path) as f:
        for line in f:
            try: out.append(json.loads(line))
            except json.JSONDecodeError: pass   # truncated tail after a crash
    return out

def try_lease(lock: Path, ttl=900) -> bool:
    if lock.exists() and time.time() - lock.stat().st_mtime < ttl:
        return False
    try:
        fd = os.open(lock, os.O_CREAT | os.O_WRONLY | (0 if lock.exists() else os.O_EXCL))
        os.close(fd); os.utime(lock, None); return True
    except FileExistsError:
        return False

def run_shard(shard: Path, res_dir: Path, backend, max_attempts=3, flush_every=200):
    """backend(list_of_items) -> list of dicts with 'id' and either 'output' or 'error'."""
    res = res_dir / (shard.stem + ".results.jsonl")
    dlq = res_dir / (shard.stem + ".dead.jsonl")
    done = {r["id"] for r in read_jsonl(res)} | {r["id"] for r in read_jsonl(dlq)}
    todo = [r for r in read_jsonl(shard) if r["id"] not in done]
    attempts = {}
    while todo:
        chunk, todo = todo[:flush_every], todo[flush_every:]
        retry = []
        for out in backend(chunk):
            if "output" in out:
                with open(res, "a") as f: f.write(json.dumps(out) + "\n")
            else:
                n = attempts[out["id"]] = attempts.get(out["id"], 0) + 1
                if n >= max_attempts:
                    with open(dlq, "a") as f: f.write(json.dumps(out) + "\n")
                else:
                    retry.append(next(x for x in chunk if x["id"] == out["id"]))
        if retry:
            time.sleep(min(30, 2 ** max(attempts[r["id"]] for r in retry)) * random.random())
            todo = retry + todo

def worker(shard_dir: Path, res_dir: Path, backend):
    res_dir.mkdir(parents=True, exist_ok=True)
    for shard in sorted(shard_dir.glob("shard_*.jsonl")):
        lock = res_dir / (shard.stem + ".lock")
        marker = res_dir / (shard.stem + ".done")
        if marker.exists() or not try_lease(lock): continue
        run_shard(shard, res_dir, backend)
        marker.touch()

Three details are worth calling out. The done set merges results and dead letters, so a resumed shard never re-attempts an item that has already exhausted its budget. The attempt counter is in memory, which is acceptable because a restart simply grants a fresh budget, but a stricter system would persist it. And the .done marker is written only after the shard finishes, which means a killed worker leaves a stale lock; another worker takes the shard once the lease TTL elapses and resumes from the done log, losing at most one flush interval of work.

Driving vLLM offline

With the runner in place, the vLLM backend is a thin adapter. The offline LLM class takes a list of prompts and a SamplingParams object and returns outputs in order. Load the model once per worker process, not per shard; load time is pure waste against your utilisation budget. Feed the engine large chunks, since the scheduler only reaches full batch when it has enough work queued, and set enable_prefix_caching=True when prompts share a prefix.

from vllm import LLM, SamplingParams

llm = LLM(model="your-org/your-model", enable_prefix_caching=True,
          gpu_memory_utilization=0.90, max_model_len=8192)
params = SamplingParams(temperature=0.0, max_tokens=256)

def vllm_backend(items):
    prompts = [SYSTEM_PREFIX + it["text"] for it in items]
    outs = llm.generate(prompts, params)
    results = []
    for it, o in zip(items, outs):
        text = o.outputs[0].text
        fin = o.outputs[0].finish_reason
        if fin == "length":
            results.append({"id": it["id"], "error": "truncated"})
        else:
            results.append({"id": it["id"], "output": text,
                            "prompt_tokens": len(o.prompt_token_ids),
                            "gen_tokens": len(o.outputs[0].token_ids)})
    return results

Treat finish_reason == "length" as a failure signal, not a result: a truncated JSON object is worse than no object. The model path, max_model_len and memory fraction above are placeholders to be tuned for your hardware; vLLM’s engine arguments evolve between releases, so confirm names against the documentation for the version you deploy.

Prefix Caching, Ordering and Structured Output at Scale

Batch workloads are unusually friendly to caching because the prompt is usually a long fixed instruction plus a short variable payload. An extraction prompt with a 1,500-token schema description and few-shot examples, followed by a 300-token document, is 83% shared prefix. Computing that prefix once instead of a million times is a large share of the prefill budget.

How vLLM’s prefix cache works

vLLM’s automatic prefix caching stores the KV cache of earlier requests and, when a new request begins with the same token sequence, reuses it instead of recomputing. Mechanically, the KV cache is divided into fixed-size blocks; each full block is identified by a hash that chains the parent block’s hash, the block’s own token IDs and any extra keys such as LoRA identifiers or a cache salt, with SHA-256 as the documented default algorithm in recent versions. Only full blocks are cached, and eviction is least-recently-used. The consequence is that sharing must be exact and at the start: a single differing token early in the prompt invalidates every block after it. Put the stable instructions first and the variable data last, and never interpolate per-row values (a timestamp, an ID) into the prefix.

The official documentation is explicit about the limit: prefix caching speeds up prefill only. It does nothing for decoding, so it helps little when outputs are long and dominate runtime, or when requests share no prefix. Run the arithmetic from the earlier worked example: prefill was 0.05 of the 0.13 GPU-seconds per item. If 83% of prefill is a cache hit, saving is about 0.041 seconds, a 32% reduction in per-item GPU time. For a classification job with 5-token outputs, the same hit rate would cut a far larger fraction. For a long-form generation job, the gain is a rounding error. These numbers are illustrative, but the shape of the conclusion is not: caching value scales with the prefill share of total time.

Provider-side caching in batches

The same logic applies on managed APIs, with weaker guarantees. Anthropic’s documentation describes caching inside batches as best-effort, with the discount stacking on top of the 50% batch discount; its recommendation to use identical cache-control blocks in every request and the one-hour cache lifetime for shared context follows directly from the prefix-matching mechanism. Because requests in a batch are not processed in strict order or at a steady rate, a cache entry can expire between hits, which is why hit rates are a range rather than a constant. Do not build a business case on the top of that range; budget for the middle and treat the rest as upside.

Ordering for the scheduler

Sorting items before submission helps both backends. For self-hosted engines, grouping by prompt length reduces the gap between the longest and shortest sequences in flight and keeps memory predictable, while grouping by shared prefix raises hit rate and keeps hot blocks from being evicted. One caution: a strict length sort puts all the longest prompts last, so the tail of the job becomes the memory-hungriest and slowest part, and an out-of-memory crash arrives hours in. A common compromise is to sort coarsely by length buckets and shuffle within buckets, or to run a short smoke pass over the longest items first to fail fast.

Structured output at scale

Most batch jobs end in a schema: labels, extracted fields, scores. Constrained decoding makes validity a property of the sampler rather than a hope. vLLM supports structured outputs through the xgrammar and guidance backends, with an automatic selection mode, and its constraint types include choice, regex, JSON schema, context-free grammar and structural tags. Offline, you pass a structured-outputs parameter object inside SamplingParams and call generate as usual. Provider batch endpoints support the equivalent through their JSON-schema response formats, subject to each provider’s schema subset.

Constrained decoding has costs worth knowing. It adds per-token masking work, which matters when you are chasing the decode ceiling, and grammar compilation for a complex schema is a one-time cost per distinct schema that you should amortise by reusing the same schema object across the job. More subtly, constraints guarantee syntax, not semantics: a schema-valid object can still contain a wrong label. And forcing an answer format can degrade reasoning-heavy tasks if the schema leaves no room for the model to think first; if you need chain-of-thought, include a free-text reasoning field before the answer field, and strip it afterwards.

Spot and Preemptible GPUs for Batch Jobs

Batch inference is the textbook workload for discounted, interruptible capacity. Providers sell spare GPU capacity at substantial discounts in exchange for the right to reclaim it, with varying notice, and interruption is exactly the failure that the shard-lease design absorbs. Discount levels and notice periods differ by cloud, region and instance family and change often, so check the current pricing and interruption documentation for your provider rather than relying on a remembered number.

Figure 4: Preemption on a spot worker. Results are flushed incrementally, the worker releases its lease on notice if it can, and a new worker resumes from the done log.

The sequence in Figure 4 has one important property: correctness never depends on the preemption notice being honoured. If the notice arrives and the worker flushes and releases the lease, the next worker starts immediately. If the node is killed without notice, the lease times out and the next worker resumes from the last durable flush. The notice is an optimisation that saves the lease TTL and one flush interval; the checkpoint is the guarantee (full-size diagram).

Several practical points decide whether spot capacity actually saves money. Model load time is dead time on every restart, so keep weights on fast local or regional storage and consider a baked machine image, because a ten-minute reload on every preemption can erase a double-digit share of the discount. Diversify across instance types and zones so that one capacity pool drying up does not stall the whole fleet. Keep the shard small enough that lost in-flight work is trivial. And model the effective price honestly: spot cost divided by (1 minus the fraction of time lost to preemption and reload). A 60% nominal discount with 15% lost time is a 53% effective one, still excellent, but the figure you should put into the break-even calculator.

Mixed fleets are a sound hedge. Run a small on-demand baseline that guarantees the job finishes by the deadline at any rate, and let spot workers pull from the same queue to accelerate it. Because the queue is the only coordination point, this requires no extra logic.

Result Validation, Observability and the Cost Calculator

A batch job without a validation gate is a machine for producing a million plausible-looking wrong rows. Validation sits between the result store and the downstream table, and it has three layers of increasing cost.

The first layer is structural: does the output parse, match the schema, and use only allowed enum values? This is free and catches truncation, refusals and format drift. The second is statistical: compare distributions against expectations, such as label frequencies, output-length percentiles and the refusal rate, per shard. A shard whose label distribution differs sharply from its neighbours usually points at a bad input segment or a mid-run prompt change. The third is semantic: score a random sample, typically a few hundred items per run, against a golden set or with a separate judge model, and gate promotion on the result. Sample by shard so that a localised problem cannot hide inside a good average.

What to measure

Track a small set of metrics per shard and per run: items completed, items dead-lettered, retries per item, prompt and generated tokens, wall-clock per shard, and for self-hosted runs the engine’s own signals, namely running and waiting request counts, KV-cache utilisation and prefix-cache hit rate. The most informative single chart in a self-hosted run is generated tokens per second over time. A healthy job plateaus near the measured peak, a flat line well below it means the engine is starved or the batch is too small, and a sawtooth means repeated restarts. For provider batches, track the batch state, the age against the 24-hour ceiling, and the counts of succeeded, errored and expired results.

Cost observability deserves its own line item. Compute cost per accepted item, not per attempted item: retries, rejected outputs and dead letters all add to the numerator while contributing nothing to the denominator. That figure is what you compare against the alternative backend, and it is routinely 10 to 30% higher than the naive tokens-times-price estimate, though the size of the gap depends entirely on your rejection and retry rates.

A break-even calculator

The calculator below encodes the worked example from earlier. Every default is an illustrative placeholder; the point is that the inputs are things you can measure: throughput from a pilot run, utilisation from a similar completed job, and prices from the current price pages.

from dataclasses import dataclass

@dataclass
class Workload:
    items: int
    in_tokens: int          # prompt tokens per item
    out_tokens: int         # generated tokens per item

@dataclass
class ApiPrice:             # post-discount batch price, USD per million tokens
    in_per_m: float
    out_per_m: float

@dataclass
class Gpu:
    usd_per_hour: float
    prefill_tps: float      # measured prompt tokens per second
    decode_tps: float       # measured generated tokens per second at chosen batch
    prefix_hit: float = 0.0 # fraction of prompt tokens served from prefix cache
    utilisation: float = 0.7  # effective, includes load, tail, retries
    spot_loss: float = 0.0  # fraction of time lost to preemption and reload

def api_cost(w: Workload, p: ApiPrice, accept_rate=1.0) -> float:
    per_item = w.in_tokens * p.in_per_m / 1e6 + w.out_tokens * p.out_per_m / 1e6
    return w.items * per_item / accept_rate

def self_host_cost(w: Workload, g: Gpu, accept_rate=1.0) -> float:
    sec = w.in_tokens * (1 - g.prefix_hit) / g.prefill_tps + w.out_tokens / g.decode_tps
    gpu_hours = w.items * sec / 3600 / (g.utilisation * (1 - g.spot_loss))
    return gpu_hours * g.usd_per_hour / accept_rate

def breakeven_utilisation(w: Workload, g: Gpu, p: ApiPrice) -> float:
    full = Gpu(**{**g.__dict__, "utilisation": 1.0, "spot_loss": 0.0})
    return self_host_cost(w, full) / api_cost(w, p)

if __name__ == "__main__":
    w = Workload(items=1_000_000, in_tokens=1000, out_tokens=200)
    p = ApiPrice(in_per_m=0.15, out_per_m=0.60)              # illustrative
    g = Gpu(usd_per_hour=2.50, prefill_tps=20_000, decode_tps=2_500,
            prefix_hit=0.0, utilisation=0.7)                # illustrative
    print(f"API:        ${api_cost(w, p):,.0f}")
    print(f"Self-host:  ${self_host_cost(w, g):,.0f}")
    print(f"Break-even utilisation: {breakeven_utilisation(w, g, p):.0%}")

With the defaults it reports about \$270 for the API, about \$129 for self-hosting at 70% utilisation, and a break-even near 33%. Now vary the inputs. Set prefix_hit=0.8 and the self-hosted cost falls by roughly a third, which moves break-even down to the low 20s. Set utilisation=0.25 for a small, bursty job and the API wins. Add spot_loss=0.15 with a lower hourly rate and watch the effective price. The exercise is more valuable than the answer: it shows which input your decision is most sensitive to, and that tells you what to measure in a pilot.

Trade-offs, Gotchas, and What Goes Wrong

Poison items and head-of-line blocking. One document that triggers a pathological generation, such as a repetition loop that runs to the token limit, occupies a batch slot for its entire length. Always set max_tokens tightly to the largest legitimate output, use a repetition-safe sampling configuration, and treat length-limited finishes as failures. Without these, a handful of runaway sequences can hold a shard open long after everything else is done.

Out-of-memory crashes mid-run. Engines reserve a memory fraction up front, but peak KV demand depends on the actual length mix. A job that survives the first two hours on short prompts can die when the long-prompt bucket arrives. Cap max_model_len at what you truly need, run a pre-flight pass on the longest items, and make the worker restart automatically; checkpointing turns an OOM from a disaster into a lost flush interval. Lowering the memory fraction slightly trades a little throughput for stability and is usually worth it for unattended runs.

Tail latency, the long end of the job. Batch jobs finish in a curve with a long tail: the last few percent of items take disproportionate wall time because the batch is no longer full and a few slow shards straggle. On self-hosted fleets, the idle GPUs during the tail are paid-for waste, so scale workers down as the queue drains and consider speculative duplicate execution of the final shards. On provider batches, the equivalent is the tail rescue described earlier: when the deadline nears and an item is unfinished, resubmit it through the online endpoint at full price. Paying double for 0.5% of rows is cheaper than missing the deadline.

Silent drift between runs. Floating-point nondeterminism, engine upgrades and provider-side model updates mean that identical inputs can produce different outputs across runs, even at temperature zero. That matters for evaluation jobs where you compare two runs. Pin the model version where the provider allows it, pin the engine version, record both in every result row, and compare runs on the same backend.

Rate-limit and quota surprises. Provider batches have their own queue-token caps, and an enqueue that exceeds them fails or waits. Expiry rules differ too: a late OpenAI or Anthropic batch returns what it completed, while an expired Gemini job returns nothing. Shard accordingly, and monitor the age of every outstanding batch against its ceiling.

The 29-day and 30-day cliffs. Results are not forever. Anthropic results are downloadable for 29 days from batch creation; OpenAI output files are deleted 30 days after completion. Copy results into your own store as soon as a batch ends, and never treat the provider as the system of record.

Data governance. Batch endpoints are still third-party processing. Check data-retention terms, regional options and whether your contracts allow prompts containing personal data to leave your environment. If they do not, the decision is made before any cost comparison begins.

Engineering cost is real cost. The calculator above has no line for the engineer who builds, debugs and babysits a GPU fleet. For a one-off job of a few hundred thousand items, the API’s fixed price and zero ops almost always win. Self-hosting pays off through repetition: the same pipeline reused across weekly evaluations, daily enrichment and quarterly re-embedding amortises the build cost.

Practical Recommendations

Start with the API unless you have a reason not to. Build the sharding, item-ID and validation layers first, because they are backend-independent and they are where most of the long-term value sits. Run the first production job on a provider batch API, keep every result row tagged with model, prompt version and cost, and use the real figures to calibrate the calculator.

Move to self-hosted vLLM when at least two of these are true: the volume is large and recurring enough that you can keep GPUs above the break-even utilisation you computed; prompts have a long shared prefix so caching changes the price materially; you need a model, fine-tune or privacy posture that no provider batch endpoint offers; or you want deterministic control over the engine version. Stay on the API when the workload is bursty, small, or one-off.

Whichever you choose, treat the first run as a pilot on a thousand items. Measure real tokens per item, real throughput, real rejection rate, and use those, not the vendor’s headline figures, in your decision.

Assign content-hash item IDs that include prompt version and model.
Shard to roughly five to fifteen minutes of work per shard, and log per-shard progress.
Append results durably and flush on a count-or-time trigger; resume from a done log.
Cap retries per item and route exhausted items to a dead-letter store.
Put stable instructions first in every prompt; enable prefix caching for self-hosted runs.
Treat length finishes as failures; use constrained decoding for schema output.
Validate structurally, statistically and semantically before promoting results.
Compute cost per accepted item and compare against the alternative backend each quarter.
Copy results out of provider storage immediately; mind the 29 and 30 day retention windows.
Keep a deadline fallback that sends the unfinished tail to the online endpoint.

Frequently Asked Questions

What is batch LLM inference?

Batch LLM inference is running many prompts through a language model as one asynchronous job rather than answering them one at a time as users wait. Because no individual response is time-critical, the system can fill GPUs to capacity or queue work during off-peak periods. That is why providers discount it by about half and why offline engines achieve far higher throughput per GPU than interactive deployments.

How much cheaper are provider batch APIs than the regular API?

OpenAI, Anthropic and Google each document a 50% discount relative to synchronous pricing for their batch endpoints, in exchange for a completion window of up to 24 hours, with Google’s target being 24 hours and expiry at 48. Anthropic notes that the discount can stack with prompt caching. Always confirm current pricing and model eligibility on each provider’s pricing page, since both change.

When does self-hosted vLLM beat a batch API on cost?

It wins when you can sustain high effective GPU utilisation and your tokens per item favour it. In the illustrative example in this post, break-even sat near one-third utilisation, and a long shared prefix pushed it lower. Small, bursty or one-off jobs rarely clear the bar once model loading, idle tails and engineering time are counted. Measure utilisation on a pilot rather than assuming it.

Does prefix caching help batch inference?

Yes, when prompts share a long identical beginning, such as instructions and few-shot examples, because the engine reuses the cached KV blocks instead of recomputing them. The vLLM documentation notes it accelerates only prefill, not decoding, so it helps most for classification or extraction with short outputs and little for long generation. Place stable text first and variable data last.

How do I make a long batch job resumable?

Give every item a stable ID, split the data into shards, and append each result to a durable log keyed by that ID. On restart, a worker reads the log, skips completed IDs and processes only the remainder. Add a lease with a timeout so that a crashed worker’s shard is reassigned, and cap retries with a dead-letter store so poison items cannot block the job.

Can I run batch inference on spot or preemptible GPUs?

Yes, and it is one of the best fits for them, provided the job checkpoints often and shards are small. Correctness should rely on the durable done log and lease expiry, not on receiving a preemption warning. Budget for model reload time and lost work when computing the effective discount, and diversify across instance types and zones to avoid capacity droughts.

Offline Batch LLM Inference Architecture: Provider Batch APIs vs Self-Hosted vLLM on Cost and Throughput (2026)