How to Handle LLM API Rate Limits: RPM and TPM Budgets, 429 Retries, Backoff with Jitter and Multi-Provider Fallback

Most teams meet LLM API rate limits the same way: a launch goes well, traffic doubles, and the logs fill with 429 Too Many Requests. The usual fix is to wrap the call in a retry decorator and move on. That fix makes things worse. Retries spend the same budget the failing calls just exhausted, and failed requests still count against your per-minute limit at OpenAI and Azure.

This matters now because limits are no longer one number. In 2026 a single provider meters requests, input tokens and output tokens separately, adds ramp limits on how fast usage may grow, and returns different status codes for “you are too fast” and “we are overloaded”. Each vendor counts tokens differently, too.

You will leave with a nine-step method, a runnable Python asyncio client that reserves token budget before each call, a retry policy that honours retry-after, and a fallback chain with a quality guard.

What this covers: how OpenAI, Anthropic, Gemini and Azure OpenAI meter traffic, the headers they return, client-side token buckets keyed by model, token estimation, full-jitter retries, idempotency, priority queues, batch APIs, provider fallback, observability and the failure modes that bite in production.

Context and Background

LLM API rate limits began as a simple requests-per-minute cap, the same model every REST API used. It changed because the cost of a request varies by four orders of magnitude. A ten-token classification call and a 200,000-token document analysis are both “one request”, so providers moved to token-based metering. Today the shape of a limit is a vector, not a scalar.

The four platforms most production teams target in late 2026 each meter differently. OpenAI’s rate limit guide lists requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), tokens per day (TPD) and images per minute, set at organization and project level and varying by model. Anthropic’s rate limit documentation meters RPM, input tokens per minute (ITPM) and output tokens per minute (OTPM) per model class, and states plainly that it uses the token bucket algorithm with continuous replenishment rather than fixed-window resets. Google’s Gemini API applies RPM, input TPM and RPD per Google Cloud project rather than per API key, with RPD resetting at midnight Pacific time. Azure OpenAI assigns TPM quota per subscription, per region, per model and per deployment type, and derives RPM from it.

The incumbent advice is “use exponential backoff”. It is correct but incomplete. Backoff is a recovery mechanism, and recovery is the expensive path. The general theory of retries, jitter and circuit breakers is covered in our guide to retries, exponential backoff, jitter and retry budgets, and the server side of throttling in distributed rate limiting with token buckets and GCRA. This post sits between the two: the client of a metered, multi-dimensional, vendor-specific API.

My thesis is simple. Treat an LLM rate limit as a budget you schedule against, not an error you recover from. The client should reserve an estimate of tokens before sending, reconcile against the real usage the response reports, and correct its local view from the provider’s headers. When you do that, 429 becomes a rare calibration signal instead of your main flow-control mechanism. The second half of the thesis is that the reservation rules must be provider-specific, because the providers disagree on what counts.

How to Handle LLM API Rate Limits: The Nine-Step Method

Direct answer: To handle LLM API rate limits, measure real token usage, then put a client-side token bucket per provider and model in front of every call that reserves estimated input and output tokens. Retry only retryable errors with full-jitter exponential backoff that honours retry-after, queue by priority, send offline work to batch APIs, and fail over to another model behind a quality guard.

The method, in order:

Measure token usage per route, model and tenant from the usage fields in responses.
Read your real limits from the provider console and the rate-limit response headers.
Put a client-side token bucket per provider and model that meters RPM, input tokens and output tokens.
Estimate tokens before each call and reserve them, then reconcile with actual usage.
Retry with full jitter, honour retry-after, and cap attempts and total time.
Make calls idempotent so a retried or duplicated request cannot double-apply side effects.
Queue with priority so interactive traffic preempts background traffic when the budget is tight.
Move offline work to batch APIs, which run against separate quotas at a discount.
Fall back across providers or models with a quality guard, and observe everything.

Figure 1: Reference architecture for rate-limit-aware LLM calls. Requests enter a priority queue, reserve budget in a per-model limiter, pass through a retry policy and a fallback router, and feed usage and headers back into the limiter and the metrics store.

The diagram reads left to right. Application requests land in a priority queue rather than calling the provider SDK directly. A dispatcher pulls the highest-priority request, estimates its tokens and asks the limiter for that model to reserve budget. Only when the reservation succeeds does the call go out. The response flows back through a reconciler that corrects the reservation with real usage and clamps the local bucket to the provider’s reported remaining budget. Errors go to the retry policy, which either schedules a delayed retry through the same limiter or hands off to the fallback router.

Why the limiter must be keyed by provider and model

Every provider in scope sets limits per model or model class. OpenAI documents that limits vary by model and that some models share a pooled limit, so a call to any model in the group draws from the same TPM. Anthropic meters per model class. Azure OpenAI quota is per deployment. A single global limiter therefore throttles the wrong thing: it will starve a cheap model because an expensive one used the budget, or let a burst on one deployment through because the global counter looks healthy.

The practical key is (provider, account or project, model-or-pool). When two models share a pool, they must share a limiter object. Get this wrong and your limiter will be confidently, consistently mistaken.

Why you need three buckets, not one

Limits trip on whichever dimension runs out first. OpenAI’s guide gives the canonical example: twenty requests of 100 tokens exhaust a 20 RPM limit long before a 150,000 TPM limit. The opposite happens with long-context traffic, where a handful of large prompts drain TPM while RPM looks idle. Anthropic separates input from output tokens, so a summarization workload (huge input, small output) and a generation workload (small input, long output) stress different buckets.

A limiter that tracks only requests, or only total tokens, will either under-use your quota or keep hitting 429. Track RPM, input TPM and output TPM as independent buckets and admit a request only when all three can cover it.

Why reservation accounting differs by provider

This is the part most write-ups miss. The providers disagree on what a request costs at admission time:

Provider	What counts toward the token limit at admission	Cache reads	Notes verified this run
OpenAI	The larger of `max_tokens` and the estimated request size from character count	Not stated in the rate-limit guide	Set `max_tokens` close to the expected response size
Anthropic	Uncached input plus cache writes for ITPM; OTPM counts only tokens actually generated	Do not count toward ITPM	`max_tokens` carries no OTPM penalty
Azure OpenAI	Actual input and generated tokens; higher max output does not reserve TPM	Not stated on the quota page	Failed requests still count toward the per-minute limit
Gemini API	Input tokens per minute plus RPM and RPD	Not stated on the rate-limit page	Limits are per project; RPD resets at midnight Pacific

The consequence is concrete. On OpenAI, an oversized max_tokens such as 8,000 on a call that usually returns 300 tokens consumes rate-limit budget as if you would generate 8,000 tokens. On Anthropic the same setting costs nothing against OTPM, but every uncached prompt token counts against ITPM. A prompt-caching strategy that achieves an 80% cache hit rate lets Anthropic traffic process roughly five times more total input per minute: the docs’ own example shows 2M ITPM supporting about 10M total input tokens per minute. Our analysis of prompt caching economics across providers covers the cost side; the rate-limit side is just as large.

So the reservation function should be a per-provider policy object, not a constant. For OpenAI, reserve max(max_tokens, estimate) against the token bucket. For Anthropic, reserve uncached input against ITPM and an expected output figure against OTPM, then reconcile. For Azure, reserve expected rather than maximum output.

Why headers are the feedback loop

A client-side bucket is a model of the server’s bucket, and models drift. Other processes share your organization’s quota, the provider may apply temporary adjustments, and your token estimates are approximate. Response headers correct the drift. OpenAI and Azure OpenAI return x-ratelimit-limit-requests, x-ratelimit-limit-tokens, x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, x-ratelimit-reset-requests and x-ratelimit-reset-tokens, and OpenAI adds project-scoped variants such as x-ratelimit-remaining-project-tokens when a project limit applies. Azure adds retry-after-ms on 429 responses.

Anthropic returns anthropic-ratelimit-requests-*, anthropic-ratelimit-tokens-*, anthropic-ratelimit-input-tokens-* and anthropic-ratelimit-output-tokens-* families with -limit, -remaining and -reset suffixes, where reset is an RFC 3339 timestamp and remaining token values are rounded to the nearest thousand. The anthropic-ratelimit-tokens-* headers report the most restrictive limit currently in force. The rule for using any of them is the same: when the server reports less remaining budget than your bucket believes, trust the server and lower your local count. Never raise it from a header, because a header from a slow response may be stale.

Walk-through: Building a Rate-Limit-Aware Client in Python asyncio

This section implements steps 1 to 9 as one small, dependency-free module. It runs offline against a simulated provider that enforces its own server-side token bucket, so you can watch the limiter work before wiring in a real SDK. The code uses only the Python standard library and was run for this article on Python 3.

Step 1: Measure token usage before you design anything

Every provider returns usage in the response body: input or prompt tokens, output or completion tokens, and on Anthropic separate cache_creation_input_tokens and cache_read_input_tokens. Log those per route, model, tenant and priority class for at least a week before choosing limiter settings. You want three numbers per route: the median and 95th percentile input size, the same for output size, and the peak requests per minute.

The output distribution matters most for OpenAI, where the reservation is driven by max_tokens. If the 95th percentile output is 600 tokens and max_tokens is 4,096, you are reserving almost seven times what you use at admission. Tightening max_tokens to just above p99 is often the cheapest throughput gain available, and it is a one-line change.

Step 2: Read the real limits, not the published table

Published tier tables are a starting point, not your number. Tier schemes also change. On this run, OpenAI’s usage-tier page listed Free, Build, Launch and Grow tiers that upgrade automatically on cumulative credit purchases of $5, $100 and $500 for the paid tiers. Anthropic’s page listed Start, Build, Scale and Custom tiers, with a possible Evaluation tier for new organizations. Gemini listed Free and Tiers 1 to 3, with Tier 2 and 3 qualification based on cumulative Google Cloud spend plus a waiting period.

Read your organization’s limits page in each console, then confirm by inspecting the -limit headers on a live response. On Azure, compare x-ratelimit-limit-tokens with the TPM you configured on the deployment: Microsoft’s quota documentation notes that a mismatch reveals a temporary adjustment is in effect.

Step 3: A client-side token bucket keyed by model

The core primitive is a continuously refilling bucket. Capacity equals the per-minute limit, and refill rate equals the limit divided by 60 per second. This mirrors what Anthropic says it does server-side, and it also matches Azure’s short evaluation window better than a fixed-minute counter would.

import asyncio, random, time
from dataclasses import dataclass, field

class TokenBucket:
    """Continuous-refill bucket. capacity = per-minute limit, refill = limit/60 per second."""
    def __init__(self, per_minute: float):
        self.capacity = float(per_minute)
        self.tokens = float(per_minute)
        self.rate = per_minute / 60.0
        self.updated = time.monotonic()

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now

    def wait_time(self, amount: float) -> float:
        self._refill()
        if self.tokens >= amount:
            return 0.0
        return (amount - self.tokens) / self.rate

    def take(self, amount: float):
        self.tokens -= amount

    def refund(self, amount: float):
        self._refill()
        self.tokens = min(self.capacity, self.tokens + amount)

    def clamp(self, remaining: float):
        """Server says we have less than we think: trust the server."""
        self._refill()
        self.tokens = min(self.tokens, remaining)


class ModelLimiter:
    """One limiter per (provider, model): RPM bucket plus input/output TPM buckets."""
    def __init__(self, rpm, itpm, otpm=None):
        self.rpm = TokenBucket(rpm)
        self.itpm = TokenBucket(itpm)
        self.otpm = TokenBucket(otpm) if otpm else None
        self.lock = asyncio.Lock()
        self.paused_until = 0.0

    async def acquire(self, est_in: int, est_out: int):
        if est_in > self.itpm.capacity:
            raise ValueError("request larger than the whole per-minute budget")
        while True:
            async with self.lock:
                pause = self.paused_until - time.monotonic()
                waits = [pause, self.rpm.wait_time(1), self.itpm.wait_time(est_in)]
                if self.otpm:
                    waits.append(self.otpm.wait_time(est_out))
                delay = max(waits)
                if delay <= 0:
                    self.rpm.take(1)
                    self.itpm.take(est_in)
                    if self.otpm:
                        self.otpm.take(est_out)
                    return
            await asyncio.sleep(min(delay, 5.0))

    def reconcile(self, est_in, est_out, real_in, real_out):
        # Give back over-reservation, charge under-reservation.
        if real_in < est_in:
            self.itpm.refund(est_in - real_in)
        else:
            self.itpm.take(real_in - est_in)
        if self.otpm:
            if real_out < est_out:
                self.otpm.refund(est_out - real_out)
            else:
                self.otpm.take(real_out - est_out)

    def pause(self, seconds: float):
        self.paused_until = max(self.paused_until, time.monotonic() + seconds)

Three design choices are worth defending. First, acquire admits a request only when all buckets can cover it, and it takes from all of them atomically under one lock, so two coroutines cannot both see enough budget and overdraw. Second, the bucket is allowed to go negative after reconciliation, which models a server that has already charged you for tokens you under-estimated; subsequent callers wait until it refills. Third, pause gives the retry policy a way to freeze admissions for a model when the server says retry-after, so that one coroutine’s 429 slows every coroutine sharing that budget.

Set the client limits at 85 to 95 percent of the server limits. The headroom absorbs estimation error and traffic from other processes on the same key. If several processes or pods share one provider key, a process-local bucket is not enough: either divide the limit by the replica count, which wastes capacity when load is uneven, or move the bucket into a shared store. The GCRA and Redis patterns in our distributed rate limiting guide apply directly, with the token estimate as the cost parameter.

Step 4: Estimate tokens before the call

The limiter needs a cost before the request goes out. There are three ways to get one, in increasing accuracy and latency. A character heuristic of roughly four characters per English token is free and good enough for budgeting, but it is badly wrong for code, non-Latin scripts and JSON. A local tokenizer for the target model family is accurate for that family and costs microseconds to milliseconds. A provider token-counting endpoint, where one exists, is exact but adds a network round trip and is itself rate limited, so reserve it for large or expensive prompts.

def estimate_tokens(text: str) -> int:
    # Crude heuristic: ~4 characters per token for English prose.
    # Replace with a real tokenizer or a provider token-counting endpoint in production.
    return max(1, len(text) // 4) + 8

The constant 8 is a deliberately rough allowance for message framing overhead; it is illustrative, not a provider-published number. What matters more than estimator precision is the reconciliation in ModelLimiter.reconcile. An estimate that is 20 percent high costs a little throughput until the refund lands. An estimate that is 20 percent low is corrected on the next response. Either way the bucket converges on the truth within one request.

For output, reserve according to the provider rule from the table above. On OpenAI, reserve max_tokens because that is what the server reserves. On Anthropic and Azure, reserve the p95 output length from Step 1, because those platforms charge only generated tokens.

Step 5: Retry with full jitter, honouring retry-after

Retries need four decisions: which errors to retry, how long to wait, how many times, and what to do with the budget in the meantime. The classification is where most bugs live.

Figure 2: Retry sequence for a 429. The client reserves budget, receives a 429 with retry-after, pauses the shared limiter, sleeps for the larger of the jittered backoff and retry-after, then re-reserves and succeeds.

The figure shows why the pause matters. Without it, one coroutine sleeps politely while fifteen others keep admitting requests into a bucket the server has just declared empty. With it, the whole model’s admission stops for the advertised interval.

Classify errors like this, based on what each provider documented on this run:

Signal	Meaning	Retry?
429 `rate_limit_error` with `retry-after` (Anthropic)	Over RPM, ITPM or OTPM	Yes, after at least `retry-after`
429 with `ramp_limit_exceeded` (Anthropic)	Usage grew faster than the ramp allows	Yes, then grow traffic gradually
429 with `enforced_spend_limit_reached` (Anthropic)	Monthly spend cap reached, no `retry-after`	No, alert a human
529 `overloaded_error` (Anthropic)	Provider-wide overload	Yes with backoff, or fail over
429 with code `slow_down` (OpenAI)	Request rate rose too quickly	Yes, reduce rate then ramp up
503 `server_is_overloaded` (OpenAI)	Model temporarily overloaded	Yes, lengthen delay if persistent
429 with `retry-after-ms` (Azure OpenAI)	Deployment TPM or RPM exceeded	Yes, after `retry-after-ms`
429 `RESOURCE_EXHAUSTED` (Gemini spend limit)	Spend-based limit in rolling window	Yes, after a short wait
400, 401, 403, 404, 413	Request or credential problem	Never

OpenAI’s guide is explicit on several points worth repeating: treat Retry-After as a minimum and add a small random delay, fall back to exponential backoff with jitter when the header is missing, cap both attempts and total retry time, and do not retry quota or billing errors. It also says that once traffic reaches 1M input TPM, increase it by no more than 50 percent every 15 minutes after a slow_down. Anthropic’s ramp-limit guidance is similar in spirit: wait at least retry-after, hold traffic at a level that succeeded, and increase gradually.

The backoff itself is “full jitter” from Marc Brooker’s AWS Architecture Blog post on exponential backoff and jitter: sleep a uniformly random time between zero and the exponential ceiling. It spreads synchronized clients across the whole window, which is what breaks thundering-herd retry waves.

class RateLimited(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

class Overloaded(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

class Fatal(Exception):
    pass

@dataclass
class Result:
    text: str
    input_tokens: int
    output_tokens: int
    headers: dict = field(default_factory=dict)

def backoff(attempt: int, base=0.5, cap=20.0) -> float:
    """Full jitter: sleep uniformly in [0, min(cap, base * 2**attempt)]."""
    return random.uniform(0, min(cap, base * 2 ** attempt))

async def call_with_retries(provider, limiter, prompt, max_out,
                            max_attempts=4, deadline_s=30.0):
    est_in = estimate_tokens(prompt)
    est_out = max_out  # pessimistic reservation; reconciled after the call
    start = time.monotonic()
    for attempt in range(max_attempts):
        await limiter.acquire(est_in, est_out)
        try:
            res = await provider.complete(prompt, max_out)
        except (RateLimited, Overloaded) as e:
            # A rejected call may still count against the per-minute limit,
            # so keep the input charge and refund only the output reservation.
            limiter.reconcile(est_in, est_out, est_in, 0)
            wait = backoff(attempt)
            if e.retry_after is not None:
                wait = max(wait, e.retry_after + random.uniform(0, 0.25))
                if isinstance(e, RateLimited):
                    limiter.pause(e.retry_after)
            if time.monotonic() - start + wait > deadline_s:
                break
            await asyncio.sleep(wait)
            continue
        limiter.reconcile(est_in, est_out, res.input_tokens, res.output_tokens)
        if "remaining-input-tokens" in res.headers:
            limiter.itpm.clamp(res.headers["remaining-input-tokens"])
        return res
    raise RateLimited()

Two details deserve attention. The retry re-enters limiter.acquire, so retries are metered like any other request and cannot exceed the budget. And the deadline is checked before sleeping, so a request never sleeps past the point where its caller has given up. A user-facing chat turn might allow 10 to 30 seconds total; a background enrichment job might allow minutes.

One more rule: if you own the retry loop, disable the SDK’s built-in retries. Anthropic’s errors page states its official SDK retries connection errors, rate limits and 5xx errors twice by default with exponential backoff, honouring retry-after, configurable via max_retries. Microsoft’s Azure OpenAI guidance likewise says to set max_retries=0 when adding your own retry library, to avoid compounding retries. Two retry layers of three attempts each is nine calls, not three.

Step 6: Make every call idempotent

A retry after a timeout is ambiguous. The provider may have finished generating, charged you and then lost the connection. If that call had side effects, such as an agent tool call that sends an email or writes a ticket, a blind retry repeats them.

Model generation itself is usually safe to repeat; it just costs money. The danger is in what your code does with the result. Give each logical operation a stable idempotency key, derived from the request ID in your system rather than from the attempt, and record completed side effects against it. Before executing a tool call or committing a write, check the key. Anthropic returns a request-id header on every response; log it alongside your own key so you can correlate a retried call with the provider’s record.

Step 7: Queue with priority instead of racing for budget

When budget is scarce, the order of admission is a product decision. A user waiting on a chat response should beat a nightly embedding backfill. A plain semaphore admits whichever coroutine wakes first.

Put an asyncio.PriorityQueue in front of the limiter with at least two classes: interactive and background. Workers pull from the queue and call acquire. Add aging, so background requests gain priority the longer they wait, or they can starve forever during a busy day. Add a queue-time deadline per class, and shed load by failing fast with a clear error once the deadline is impossible, rather than letting requests time out deep in the stack. Admission control with explicit shedding is kinder to users than a spinner that ends in a 504.

Step 8: Send offline work to batch APIs

Any work that does not need an answer within seconds should not compete with interactive traffic at all. Batch APIs move it to a separate quota and price it lower. OpenAI’s Batch API documentation, checked this run, states a 50 percent discount versus synchronous APIs, a 24-hour completion window, up to 50,000 requests and 200 MB per input file, and a separate rate-limit pool so batch usage does not consume per-model synchronous limits. Anthropic’s Message Batches API, announced on 8 October 2024, offers 50 percent off input and output tokens and processes each batch in under 24 hours; its rate-limit page lists batch queue limits by tier and notes that batches are not subject to ramp limits. Gemini’s batch mode has its own per-model enqueued-token limits by tier.

The architectural point is that batch quota is additive. Moving a nightly classification job to batch frees its entire share of synchronous TPM for daytime users. For scheduling patterns, see our deep dive on offline LLM batch inference and batch APIs.

Step 9: Fall back across providers and models with a quality guard

Fallback is the last line of defence, not the first. It earns its place for provider-wide overloads such as Anthropic’s 529 or OpenAI’s 503 server_is_overloaded, regional incidents, and the case where your own budget is exhausted for longer than the user can wait.

Figure 3: Fallback decision flow. Retryable errors retry on the primary within a deadline; overloads and exhausted budgets move to the next provider; every candidate response passes a quality guard before it is returned.

Each hop in the chain has its own limiter, because each provider and model has its own budget. A request that falls through to the secondary reserves budget there like any other request, which prevents the classic failure where the whole fleet fails over at once and immediately exhausts the backup.

async def call_with_fallback(chain, prompt, max_out, quality_ok=lambda r: True):
    last = None
    for provider, limiter in chain:
        try:
            res = await call_with_retries(provider, limiter, prompt, max_out)
            if quality_ok(res):
                return res
        except (RateLimited, Overloaded) as e:
            last = e
    raise last or Fatal("no provider produced an acceptable answer")

The quality_ok hook is the guard. Models are not interchangeable: a fallback model may ignore your JSON schema, produce a different tone, or fail a task your evals passed on the primary. Minimum guards are schema validation for structured output, a length sanity check and a refusal check. For high-stakes routes, keep a per-route allow-list of fallback models that have passed your offline eval suite, and refuse to fall back to anything else. Log which model served every response, so product analytics can separate quality changes from model changes.

Running the whole thing

The simulated provider below enforces its own server-side bucket and injects random overloads so the example exercises every path.

class FakeProvider:
    """Simulates a server-side token bucket so the example runs offline."""
    def __init__(self, name, rpm, itpm, overload_p=0.0):
        self.name = name
        self.server = ModelLimiter(rpm, itpm)
        self.overload_p = overload_p
        self.calls = 0
        self.rejections = 0

    async def complete(self, prompt: str, max_out: int) -> Result:
        self.calls += 1
        await asyncio.sleep(0.02)
        if random.random() < self.overload_p:
            self.rejections += 1
            raise Overloaded(retry_after=0.5)
        n_in = estimate_tokens(prompt)
        if self.server.rpm.wait_time(1) > 0 or self.server.itpm.wait_time(n_in) > 0:
            self.rejections += 1
            ra = max(self.server.rpm.wait_time(1), self.server.itpm.wait_time(n_in))
            raise RateLimited(retry_after=round(ra, 2))
        self.server.rpm.take(1)
        self.server.itpm.take(n_in)
        out = random.randint(max_out // 4, max_out)
        return Result(f"{self.name}-ok", n_in, out,
                      {"remaining-input-tokens": self.server.itpm.tokens})

async def main():
    random.seed(7)
    primary = FakeProvider("primary", rpm=120, itpm=20_000, overload_p=0.15)
    secondary = FakeProvider("secondary", rpm=120, itpm=20_000)
    # Client-side limits sit a little BELOW the server's, leaving headroom.
    chain = [(primary, ModelLimiter(rpm=110, itpm=18_000, otpm=40_000)),
             (secondary, ModelLimiter(rpm=110, itpm=18_000, otpm=40_000))]
    prompts = ["Summarise this maintenance log entry. " * random.randint(5, 40)
               for _ in range(120)]
    sem = asyncio.Semaphore(16)

    async def one(p):
        async with sem:
            return await call_with_fallback(chain, p, max_out=300)

    t0 = time.monotonic()
    results = await asyncio.gather(*(one(p) for p in prompts), return_exceptions=True)
    ok = [r for r in results if isinstance(r, Result)]
    print(f"completed={len(ok)}/{len(prompts)} in {time.monotonic()-t0:.1f}s")
    print(f"served_by_primary={sum(r.text.startswith('primary') for r in ok)} "
          f"served_by_secondary={sum(r.text.startswith('secondary') for r in ok)}")
    print(f"primary calls={primary.calls} rejections={primary.rejections}; "
          f"secondary calls={secondary.calls} rejections={secondary.rejections}")

if __name__ == "__main__":
    asyncio.run(main())

On the run made for this article, the output was:

completed=120/120 in 45.9s
served_by_primary=118 served_by_secondary=2
primary calls=141 rejections=23; secondary calls=2 rejections=0

Read this as a behaviour check, not a benchmark; the provider is simulated and the numbers depend on the random seed. The workload asked for roughly 24,000 input tokens against an 18,000 per-minute client budget, so the limiter spread admissions over about 46 seconds instead of firing everything at once. An instrumented rerun showed all 23 rejections were injected overloads: the simulated server never returned a budget 429, because the client bucket admitted nothing it could not cover. Two requests exhausted their retries on the overloaded primary and were served by the secondary, through its own limiter.

Observability: the signals that tell you the limiter is right

Step 9 ends with “observe everything”, and that deserves specifics. Emit these per provider and model: limiter wait time as a histogram, reservations versus reconciled actual tokens, 429s split by error code, 5xx and overload counts, retries per logical request, fallback rate, and queue depth per priority class. Record the remaining headers as gauges. The OpenTelemetry generative-AI semantic conventions give you standard attribute names for model and token usage; our walkthrough of OpenTelemetry GenAI conventions and token cost attributes shows how to wire them into a collector.

For LLM API rate limits, three ratios do most of the diagnostic work. Retries per request above roughly 1.1 means the limiter is too generous or something else shares your key. A reservation-to-actual ratio far above 1 means your estimator or max_tokens is wasting capacity. Limiter wait time rising while 429s stay at zero means you are correctly saturated, which is a capacity-planning conversation, not a bug. Those thresholds are my working heuristics, not provider guidance.

Trade-offs, Gotchas, and What Goes Wrong

Handling LLM API rate limits on the client is not free. It adds latency by design, since it makes requests wait locally instead of failing fast at the server. It is only as good as its configuration, and it is a process-local view of a shared, organization-wide budget.

Figure 4: Two failure modes. Left, layered retries multiply load on an already saturated provider. Right, a stream that fails after the 200 status needs its own recovery path because status-code retry logic never sees it.

Retry storms. Retries amplify across layers. If an agent framework retries three times, the SDK retries twice per attempt, and a gateway retries again, one user action can produce a dozen or more upstream calls during exactly the minute the provider is saturated. Since failed requests still count toward per-minute limits on OpenAI and Azure, the storm deepens the hole it is trying to climb out of. The cure is one retry owner per call path, a retry budget that caps retries as a fraction of total traffic, and a circuit breaker that stops calling a model during sustained overload. Our retries and retry budgets guide covers the budget math.

Partial streams. Streaming changes the failure contract. Anthropic’s errors page notes that with server-sent events an error can arrive after the API has already returned 200, and that standard status-code handling does not apply. Your retry wrapper never sees a 429; it sees a stream that stops or emits an error event midway. You must decide whether to discard the partial output and retry the whole request, which is safe but charges twice, or to surface the partial text with a “generation interrupted” marker. Never silently concatenate a retried stream onto a partial one: the second generation will not continue the first.

Ramp limits surprise batch-style bursts. Both OpenAI’s slow_down code and Anthropic’s ramp limits reject sharp increases even under your nominal TPM. A cron job that wakes up and launches 2,000 requests in the first second will hit them. Smooth the start with a warm-up period that raises the limiter’s refill rate over several minutes.

Shared keys and noisy neighbours. If staging, a data science notebook and production share one organization or project, your limiter sees only its own traffic. Separate projects or workspaces per environment, and use the -remaining headers to clamp.

Per-minute limits can be enforced per second. Anthropic notes that 60 RPM may be enforced as one request per second, and Azure evaluates rates over windows of typically 1 or 10 seconds, so a 600 RPM deployment rejects an 11-request burst in one second. A limiter configured with a full minute of burst capacity can therefore still trip short-window enforcement. If you see 429s with plenty of minute-level budget left, cap burst size at a few seconds of refill.

Fallback hides regressions. A fallback that fires quietly can change answer quality for hours. Alert on fallback rate, not just on errors.

Spend caps look like rate limits. Anthropic’s spend-cap 429 carries no retry-after and will keep failing until the next month or a tier change; OpenAI also returns 429 for hard spend limits. Retrying these is pointless and noisy. Route them to an alert.

Practical Recommendations

The best way to handle LLM API rate limits is to start with measurement and headroom, not with a retry decorator. The order of work that pays back fastest is: tighten max_tokens where the provider reserves it, separate projects per environment, add a per-model limiter at about 90 percent of the published limit, and give it a single retry owner. Only then add fallback, because fallback without the other pieces mostly moves the outage to a second provider.

If you run more than a handful of services against LLM providers, centralize this logic. A shared gateway enforces budgets per tenant, owns retries, and holds the fallback chain in one place. Our comparison of LLM gateways including LiteLLM, Portkey, Kong AI Gateway and Cloudflare AI Gateway covers which ones implement token-aware limits and provider fallback. Keep a thin client-side limiter even behind a gateway, so one runaway service cannot consume another’s share.

For capacity, plan on the binding dimension. Compute for each route the requests, input tokens and output tokens per minute at peak, divide each by the corresponding limit, and the largest ratio is your utilization. If it is above about 70 percent at peak, request higher limits or move work to batch before launch, not after.

Checklist:

[ ] Usage logged per route, model and tenant, including cache reads and writes.
[ ] Limiter keyed by provider, project and model or shared pool, with RPM, input TPM and output TPM buckets.
[ ] Provider-specific reservation rule: max_tokens on OpenAI, expected output on Anthropic and Azure.
[ ] Client limits at 85 to 95 percent of server limits; clamp from -remaining headers.
[ ] Exactly one retry owner; SDK retries disabled when you own the loop.
[ ] Full-jitter backoff, retry-after treated as a minimum, attempt and deadline caps.
[ ] Spend-cap and 4xx errors never retried.
[ ] Idempotency keys on any call that triggers side effects.
[ ] Priority queue with aging and load shedding.
[ ] Offline workloads on batch APIs.
[ ] Fallback chain with per-hop limiters, a quality guard and a fallback-rate alert.

Frequently Asked Questions

What does 429 Too Many Requests mean on an LLM API?

HTTP 429 means the provider refused the request because a limit was exceeded. On LLM APIs that limit may be requests per minute, input or output tokens per minute, daily quotas, a ramp limit on how fast usage grows, or a spend cap. The response body tells you which: Anthropic uses rate_limit_error with codes such as ramp_limit_exceeded or enforced_spend_limit_reached, and OpenAI uses a slow_down code for rapid rate increases. Retry rate-limit 429s after the retry-after interval; never retry spend-cap 429s, which need a human or a billing change.

How is a tokens per minute limit calculated?

It depends on the provider. OpenAI calculates the rate-limit cost from the larger of max_tokens and the estimated size of your request, so a large max_tokens consumes budget even if the model writes little. Anthropic counts uncached input tokens and cache writes toward input TPM, excludes cache reads, and counts only actually generated tokens toward output TPM. Azure OpenAI uses actual input and generated tokens and says higher max output values do not reserve TPM. Gemini meters input tokens per minute per project. Always confirm against your own console and headers.

What is the best LLM retry backoff strategy?

Use exponential backoff with full jitter: wait a uniformly random time between zero and a ceiling that doubles each attempt, capped at a maximum. If the response includes retry-after or retry-after-ms, treat it as a minimum and add a small random delay. Cap both the number of attempts and the total elapsed time to fit the caller’s deadline. Pause the shared limiter for that model when the server sends retry-after, so other workers stop too. Make sure only one layer retries, otherwise SDK, framework and gateway retries multiply.

How do I set up LLM provider fallback without hurting quality?

Give every hop in the fallback chain its own rate limiter, because each provider and model has its own budget. Fall back on overloads, regional failures or budgets exhausted beyond the caller’s deadline, not on every 429. Pass each fallback response through a quality guard: schema validation for structured output, length and refusal checks, and an allow-list of models that passed your offline evaluations for that route. Log which model served each request and alert on fallback rate, so a quiet failover does not mask a quality regression for hours.

Do cached prompt tokens count toward rate limits?

On Anthropic, cache reads do not count toward input tokens per minute, while cache writes and uncached input do. Anthropic’s documentation gives an example where an 80 percent cache hit rate lets a 2M ITPM limit process about 10M total input tokens per minute. That makes prompt caching a throughput tool as well as a cost tool. OpenAI, Azure OpenAI and Gemini did not state cached-token treatment on the rate-limit pages checked for this article, so verify on your own traffic by comparing usage fields with the remaining headers.

Should I use a batch API to avoid rate limits?

Yes, for any work that can wait hours. OpenAI’s Batch API runs on a separate rate-limit pool from synchronous calls, at a 50 percent discount, with a 24-hour completion window. Anthropic’s Message Batches API is also discounted by 50 percent, finishes batches within 24 hours and is not subject to ramp limits. Gemini batch mode has its own enqueued-token limits. Moving nightly enrichment, evaluation runs and backfills to batch frees synchronous budget for users, and it removes the burst patterns that trigger ramp limits.

How to Handle LLM API Rate Limits: RPM and TPM Budgets, 429 Retries, Backoff with Jitter and Multi-Provider Fallback (2026)