Circuit Breaker Pattern, Retries, Backoff with Jitter and Retry Budgets: Resilience Patterns (2026)

Retries feel like free reliability. A call fails, you try again, and the user never notices. Then one day a database slows down for thirty seconds, every client in the fleet retries three times, the offered load triples exactly when capacity is lowest, and the outage outlives its own trigger by an hour. The code that was supposed to hide the failure became the failure.

The circuit breaker pattern is one of four cooperating controls that prevent this: capped exponential backoff with jitter spreads retries in time, a retry budget caps how many retries the fleet may add, a circuit breaker stops calls to a dependency that is clearly down, and bulkheads stop one slow dependency from eating every thread. Used alone, each has a hole. Used together with deadlines and load shedding, they turn a retry storm into a bounded, recoverable event.

What this covers: why naive retries cause metastable failures, the amplification arithmetic, the three jitter variants, retry budgets in gRPC and Envoy, hedged requests, the breaker state machine and its tuning, bulkheads, deadlines, idempotency, a runnable simulation, the library landscape and the failure modes of the protections themselves.

Context and Background

Every network call in a distributed system can fail in ways the caller cannot distinguish: the request was lost, the server crashed, the server is slow, or the response was lost after the work was done. Retrying is the obvious response, and for transient faults it is correct. A single dropped packet or a one-second leader election is exactly what a retry hides.

The trouble is that transient and persistent faults look identical at the moment of failure. A timeout could mean a blip or an overloaded dependency that needs less traffic, not more. A retry policy that is right for the first case is catastrophic in the second, and a dependency under overload is the situation where you can least afford extra load.

Researchers call the resulting pattern a metastable failure. In the HotOS 2021 paper by Bronson and colleagues, a metastable failure is one where a trigger such as a load spike or deployment pushes the system into a bad state that persists even after the trigger is removed, sustained by a feedback loop. Marc Brooker’s summary of the idea names retries as a classic sustaining effect: the system keeps working, but clients have already given up, so most of the work is wasted while the retries keep the queue full.

The industry response has converged on a layered toolkit. Google’s SRE book on handling overload describes per-request retry limits and per-client retry budgets. The gRPC retry design specifies throttling with a token bucket. Envoy exposes retry budgets and circuit-breaker thresholds as first-class cluster settings. Libraries such as Resilience4j and Polly package the same ideas for application code.

This article treats those controls as one system rather than five separate features. If you are also designing the traffic-admission side, read our guide to distributed rate limiting with token bucket and GCRA, because rate limiting and retry control are two halves of the same load-management problem. For the structural side of containing damage, see cell-based architecture and blast radius.

One caution on vocabulary. Throughout this post “success rate” and “goodput” mean replies the client was still waiting for, not requests the server merely completed. That distinction is the entire reason retry storms are survivable only with explicit controls.

Why Retries Cause a Retry Storm: Amplification Math

A retry storm is a feedback loop in which failures generate retries, retries generate more load, and more load generates more failures. To see how severe it can get you only need multiplication, and the multiplication compounds with call-chain depth.

Per-layer retries multiply

Suppose a request passes through four services in a chain, A calls B, B calls C, C calls D, and D calls a database. Each layer is configured, reasonably in isolation, to make up to three attempts. If the database is failing, the worst-case number of database calls for one user request is 3 x 3 x 3 x 3 x 3 = 243 when the user-facing client also retries, or 81 for the four service layers alone. Figure 1 shows the shape.

Figure 1: Retries multiply at each layer. Three attempts per hop turns one request into 27 calls at service D and 81 at the database.

The arithmetic is simple, but nobody sees it because each team owns one hop and each hop’s setting looks modest. The database only sees an 81x spike on a request that would have failed anyway. Google’s SRE guidance is blunt about the rule that follows: a failed request from the lowest layer should be retried only by the layer immediately above it, and upper layers should receive an error that says “do not retry” rather than retrying again.

Overload makes the retries worse than the original failures

Amplification is bad when failures are independent. It is much worse when they are load-dependent. If a backend’s capacity is 1,000 requests per second and the offered load is 800, a short capacity dip produces failures, and those failures return as retries on top of the 800. Offered load might be 1,400, which fails more requests, which retry, and so on. The loop is positive feedback with a gain greater than one.

There is a second, quieter cost: wasted work. When a client times out after one second and sends a retry, the server still has the original request in its queue. It will process the stale request, produce a response nobody reads, and only then reach the retry. Capacity is spent on answers with no audience.

What the SRE numbers say

The SRE book quantifies the effect of limiting retries at the client. With retries allowed freely and a datacenter rejecting most requests, traffic can approach a factor “just below 3X” of the original attempted rate under a three-attempt policy. Adding a per-client rule that retries continue only while retries are under 10% of requests cuts the growth to about 1.1x in the general case. That 10% figure is Google’s published example, not a universal constant, but it shows the order of magnitude: a budget is worth roughly a factor of three of protection even in the simple case, and far more in deep chains.

The simulation

To make the dynamics concrete, the repository for this post includes a small discrete-time simulation, shown in full later. The backend serves 1,000 requests per second FIFO, the offered load is 800 per second, a trigger cuts capacity to 40% for five seconds, and clients abandon requests older than one second. The numbers below come from running that script; they illustrate the mechanism and are not a benchmark of any real system.

Policy	Peak offered load (rps)	Avg goodput, t=60 to 89 s (rps)	Queue at t=89 s
No retries	800	800	0
Naive retries, up to 4 attempts	3,200	0	151,000
Retry budget (10% of good replies)	900	800	0
Budget plus breaker	900	800	0

Read the second row carefully. The trigger lasted five seconds. Sixty seconds after it ended, naive retries still deliver zero goodput because the queue is full of stale work. That is a metastable failure in about 60 lines of Python. The no-retry policy recovers by itself the moment capacity returns, which is the uncomfortable lesson: for this load shape, the retries are strictly worse than doing nothing.

Exponential Backoff with Jitter: The Reference Retry Loop

Exponential backoff with jitter means waiting an exponentially growing, capped interval between attempts and then randomizing that wait so that clients that failed together do not retry together. Backoff reduces the average retry rate; jitter removes the synchronization that turns many independent clients into one coordinated wave.

Figure 2 shows the decision path a well-behaved client walks on every failure. Note that the sleep is the last step, not the first. Most of the protection comes from the gates in front of it.

Figure 2: A retry is allowed only if the error is retryable, the operation is idempotent, attempts and deadline remain, and the retry budget has tokens. Only then does exponential backoff with jitter pick a sleep.

The flow reads left to right. The first gate is whether the failure is retryable at all: INVALID_ARGUMENT or an HTTP 400 will fail identically forever. The second gate is whether the call is safe to repeat, which we return to under idempotency. The third gate is attempts and remaining deadline. The fourth is the budget. The sleep only happens after all four.

Why plain exponential backoff is not enough

Without jitter, clients that failed at the same instant all compute the same delays: 100 ms, 200 ms, 400 ms and so on. The load on the server is a series of sharp synchronized spikes at 100 ms, 300 ms, 700 ms after the failure. Each spike is as tall as the original burst. Backoff has made the retries rarer but not smaller, and a server that could not take the burst the first time will not take it the third time either.

The classic analysis is the AWS Architecture Blog post “Exponential Backoff and Jitter” by Marc Brooker, which simulated many clients contending for a resource and compared the total calls and completion time of each approach. It reported that all jittered variants cut the work substantially compared with no jitter, that full and equal jitter needed about the same number of calls with equal jitter doing slightly more, that decorrelated jitter needed more calls, and that no-jitter backoff took so long to finish that it was left off the comparison graph. The post is the origin of the three standard variants below. I am quoting its qualitative findings only; the exact call counts depend on its simulation parameters, which you should read at the source.

The three variants

With base as the initial delay, cap as the maximum delay and attempt counting from zero, the standard formulations are:

Full jitter: sleep = random(0, min(cap, base * 2^attempt)). The wait is uniform over the whole capped interval, so some clients retry almost immediately and the load spreads smoothly.
Equal jitter: temp = min(cap, base * 2^attempt); sleep = temp/2 + random(0, temp/2). Half the delay is guaranteed, half is random. It prevents very short sleeps but keeps clients clustered in the upper half, so it takes longer to drain.
Decorrelated jitter: sleep = min(cap, random(base, previous_sleep * 3)). The ceiling grows from the last random value rather than from the attempt count, so clients decorrelate from each other over time.

Which to pick? Full jitter is a good default: lowest total work among the variants in the AWS analysis, and the simplest to reason about. Decorrelated jitter is attractive when you want to avoid synchronization after a very long outage, at the cost of more calls. Equal jitter is rarely the best choice when the goal is to minimize load.

A reference implementation

The following Python is the retry loop in Figure 2 with a deadline, full jitter and a pluggable budget. It is deliberately small so you can read the control flow.

import random, time

class RetryableError(Exception): ...

def call_with_retries(fn, *, base=0.1, cap=5.0, max_attempts=4,
                      deadline_s=8.0, budget=None):
    start = time.monotonic()
    for attempt in range(max_attempts):
        try:
            return fn(timeout=max(0.05, deadline_s - (time.monotonic() - start)))
        except RetryableError:
            remaining = deadline_s - (time.monotonic() - start)
            last = attempt == max_attempts - 1
            if last or remaining <= 0:
                raise
            if budget is not None and not budget.try_acquire():
                raise                      # fail fast instead of adding load
            sleep = random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(min(sleep, remaining))
    raise AssertionError("unreachable")

Two details matter. The per-attempt timeout is derived from the remaining deadline, so a retry never outlives the user’s patience. And a refused budget re-raises the original error immediately rather than sleeping, which is what makes the budget a load-shedding tool and not just a delay.

What the gRPC design pins down

gRPC’s client-retry proposal is a useful reference because it is explicit. Its retry policy has maxAttempts, which counts the original call and is silently capped at 5, plus initialBackoff, maxBackoff, backoffMultiplier and a non-empty list of retryableStatusCodes. The first retry waits initialBackoff times a random factor between 0.8 and 1.2; the nth retry waits min(initialBackoff * backoffMultiplier^(n-1), maxBackoff) times the same factor. That is exponential backoff with a narrow jitter band, not full jitter, which is a defensible choice for a single-hop client but is weaker against large synchronized fleets than full jitter. The call deadline applies across all attempts, which is the right semantics. The details are in the gRPC A6 proposal.

Retry Budgets: Capping the Fleet, Not the Request

A retry budget limits retries as a proportion of normal traffic, so that the total extra load a fleet can generate is bounded no matter how many requests fail. A per-request limit of three attempts bounds one request. A budget bounds the aggregate, which is the quantity that actually damages a dependency.

The key property is that a budget is proportional to success. If everything is healthy, the budget is full and an occasional failure is retried. If the dependency collapses, successes stop, the budget drains, and retries stop. The system sheds exactly the load that was making things worse, without an operator in the loop.

gRPC retry throttling

gRPC implements this as a token bucket per server name. The bucket starts at maxTokens (an integer up to 1000). Each failed RPC removes one token and each successful RPC adds tokenRatio tokens. When the count falls to half of maxTokens or lower, retries and hedged RPCs are not sent until the count climbs above that threshold. Only retryable or non-fatal failures count; an INVALID_ARGUMENT error does not drain the bucket.

Take maxTokens = 10 and tokenRatio = 0.1, an illustrative configuration. The bucket starts full at 10. Five consecutive failures bring it to 5, which is at the threshold, so retries stop. A single success adds 0.1, lifting the count to 5.1, so retries resume, but every failed call removes a whole token while each success restores only a tenth of one. The effect is a steady-state ceiling: the bucket drifts down whenever the failure fraction exceeds roughly tokenRatio / (1 + tokenRatio), about 9% here, and then hovers at the threshold with most retries suppressed.

Envoy retry budgets

Envoy’s cluster circuit-breaker configuration includes a RetryBudget with budget_percent, which defaults to 20%, and min_retry_concurrency, which defaults to 3. The budget is a percentage of active requests that may be retries, with a floor so low-traffic clusters can still retry a little. This is a concurrency-based budget rather than a token bucket, but the intent is identical: retries are a bounded fraction of live traffic. Envoy’s older max_retries threshold, which defaults to 3 concurrent retries per cluster in the thresholds, is a hard cap for the same purpose. See the Envoy circuit breaker reference for the exact semantics and for which setting takes precedence.

Per-client and per-request, together

Google’s SRE guidance combines two limits: at most three attempts per request, and a per-client rule that retries continue only while the retry ratio is below 10%. These are complementary. The per-request limit stops one unlucky request from hammering; the per-client ratio stops the fleet. Pair both with a backend signal. When a backend is overloaded it can return a distinct “overloaded, do not retry” error so that no layer above retries, which collapses the multiplicative chain in Figure 1 to a single layer of retry.

Hedged Requests: Retrying Before the Failure

A hedged request sends a second copy of a request to another replica when the first has not answered within a chosen delay, and uses whichever reply arrives first. It attacks tail latency, not errors, and it is retry-like enough to deserve the same budget.

Dean and Barroso’s “The Tail at Scale” shows why tail latency matters: in a fan-out of many parallel calls, the slowest call sets the response time. In their BigTable benchmark, reading 1,000 keys spread across 100 servers, sending a hedged request after a 10 ms delay reduced the 99.9th-percentile latency for retrieving all values from 1,800 ms to 74 ms, while sending only 2% more requests. The paper also suggests hedging after the 95th-percentile latency as a way to bound the extra load.

gRPC’s hedging policy follows the same shape. The first RPC goes out immediately; each additional hedge goes out after hedgingDelay with no response; any successful response cancels the others; a nonFatalStatusCodes response triggers the next hedge at once. A hedging policy and a retry policy cannot both be set on one RPC, and hedges are subject to the same throttling bucket as retries. That last point is the safeguard: if the cluster is unhealthy, hedging turns itself off.

Hedging is only safe for idempotent reads, or for writes protected by an idempotency key, because two copies may both execute. It is also worthless against overload. If every replica is saturated, the hedge is just more traffic.

The Circuit Breaker Pattern: State Machine and Tuning

The circuit breaker pattern wraps a remote call in a small state machine that stops sending traffic to a dependency once it looks unhealthy, and probes carefully before resuming. Where a retry budget limits how much extra load you add, the breaker removes load entirely for a while, which gives the dependency room to recover and gives your callers a fast, cheap failure instead of a slow one.

Figure 3 shows the three states. The names come from the electrical device: closed means current flows, open means it does not.

Figure 3: The circuit breaker pattern as a state machine. Closed counts outcomes, open rejects fast, half open lets a few trial calls decide the next state.

In the closed state calls pass through and each outcome is recorded in a sliding window. When the failure rate or the slow-call rate crosses its threshold, the breaker moves to open. In the open state calls are rejected immediately without touching the network, which in Resilience4j surfaces as a CallNotPermittedException. After a wait duration the breaker moves to half open, where a limited number of trial calls are allowed. If they succeed it closes; if the failure rate is over the threshold again it reopens.

Failure rate versus slow-call rate

A breaker that only counts errors misses the most common real failure: a dependency that still answers, but slowly. Slow responses hold threads, connections and memory in the caller, and they eventually cause the caller to time out and retry. Resilience4j therefore tracks two rates, failureRateThreshold and slowCallRateThreshold, with a slowCallDurationThreshold that defines what counts as slow. Counting slowness as failure lets the breaker open before the caller’s pools are exhausted.

The documented Resilience4j defaults make the tuning problem concrete. failureRateThreshold is 50%, slowCallRateThreshold is 100% (effectively disabled until you lower it), slowCallDurationThreshold is 60,000 ms, slidingWindowType is count-based with a slidingWindowSize of 100, minimumNumberOfCalls is 100, waitDurationInOpenState is 60,000 ms and permittedNumberOfCallsInHalfOpenState is 10. Those are library defaults, not recommendations. A 60-second slow-call threshold is longer than most user timeouts, and a minimum of 100 calls means a low-traffic endpoint never trips at all.

Polly, the .NET resilience library, uses different names for the same ideas. Its circuit breaker strategy defaults to a FailureRatio of 0.1, a MinimumThroughput of 100, a SamplingDuration of 30 seconds and a BreakDuration of 5 seconds, and it ignores the failure ratio entirely if minimum throughput is not reached in the window. Polly also offers an Isolated state, a manually held open circuit, which Resilience4j calls FORCED_OPEN. The two libraries disagree by a factor of five on the default failure threshold, which is a reminder that defaults encode each library’s assumptions, not truths about your service.

Count-based versus time-based windows

A count-based window looks at the last N calls. It adapts to traffic: at 5 requests per second a window of 100 spans 20 seconds, at 5,000 requests per second it spans 20 milliseconds. A time-based window looks at the last N seconds, which gives a stable reaction time but can be dominated by a handful of calls at night. Count-based windows suit services with steady traffic and a need for quick trip decisions; time-based windows suit bursty traffic where a fixed observation period is easier to reason about. In both cases the minimum-calls guard matters, because without it one failure in the first two calls is a 50% failure rate.

Breaker per dependency, not per process

Scope is the part teams most often get wrong. One breaker around a whole client library means a failure in one endpoint opens the circuit for all of them. One breaker per host means a single bad replica trips a breaker while healthy replicas sit idle. A common compromise is a breaker per logical dependency and operation class, combined with host-level ejection from the load balancer.

That host-level mechanism exists in Envoy as outlier detection. By default, five consecutive 5xx responses (consecutive_5xx) cause a host to be ejected, the analysis interval is 10 seconds, the base_ejection_time is 30 seconds multiplied by the number of times that host has been ejected, and at most 10% of hosts (max_ejection_percent) can be ejected at once. The last number is the key design idea: ejection must never remove the whole pool. An optional max_ejection_time_jitter spreads re-admission so every proxy does not return to a host in the same instant. Istio exposes the same mechanism through its destination rule settings, so verify the field names against your Istio version.

Half-open: where breakers hurt you

The half-open state is the weakest point of the design. If the breaker allows only a few trial calls, the sample is tiny; one lucky success closes the circuit and releases full traffic to a dependency that has not recovered, producing a thundering herd that trips the breaker again. This flapping is common. If the breaker admits too many trial calls, the probe itself overloads the dependency.

Mitigations are straightforward. Require several consecutive successes, not one. Ramp traffic back in stages rather than snapping from zero to full. Jitter the open duration so that a fleet of instances does not probe simultaneously; fleet-wide synchronized probes are the retry storm in a different costume. And remember the default in Resilience4j that the open-to-half-open transition happens when a call arrives after the wait duration, unless you enable automatic transition, so an idle service re-probes lazily.

Bulkheads, Deadlines and Load Shedding

Retries and breakers are reactive. The remaining controls are structural: they limit how far a failure can spread before any detector fires.

The bulkhead pattern

The bulkhead pattern takes its name from the watertight compartments in a ship’s hull. A failing dependency is allowed to flood only its own compartment. In software, a compartment is a dedicated pool of threads, connections or concurrency permits per dependency.

Without bulkheads, a shared worker pool of 200 threads serves calls to a payment service and a recommendation service. If recommendations slow to 30 seconds per call, then at 10 requests per second about 300 threads are demanded, the pool is exhausted, and payments, which are perfectly healthy, start to time out. With a bulkhead capping recommendations at 20 concurrent calls, the 21st recommendation call fails immediately and 180 threads remain for payments. The arithmetic is Little’s law: concurrency equals arrival rate times latency, so a latency spike turns directly into a concurrency spike unless something caps it. Our post on cell-based architecture and blast radius applies the same idea at the scale of whole stacks.

Semaphore bulkheads cap concurrent calls on the caller’s thread and cost almost nothing. Thread-pool bulkheads isolate execution on separate threads and also provide a timeout boundary, at the price of context switches and queueing. Prefer semaphores with explicit timeouts unless the library you call blocks without any timeout support.

Deadlines and timeout propagation

A timeout is local; a deadline is global. If a user request has an 8-second deadline, each hop should receive the remaining time, not a fresh 8 seconds. Otherwise a service that retries three times with per-call timeouts of 8 seconds can run for 24 seconds while the user has long gone.

gRPC propagates deadlines natively: the client’s deadline travels with the call and applies across all retry attempts, so a retry loop cannot extend it. In HTTP systems you pass a header such as an absolute timestamp or a remaining-milliseconds value and every service reads it. The critical rule is to subtract expected downstream time before calling. If fewer than the minimum useful milliseconds remain, fail immediately instead of starting work that cannot finish. This turns stale-work waste, the mechanism in the simulation, into an early cheap rejection.

Load shedding on the server side

The server’s contribution is to refuse work it cannot finish. A concurrency limiter that rejects with a fast 503 or RESOURCE_EXHAUSTED once in-flight requests exceed a threshold keeps latency bounded for the requests it accepts. Pair it with the “do not retry” signal described earlier, and with the queue discipline of dropping the oldest requests first when the queue is full, because the oldest requests are the ones whose clients have most likely given up. Last-in-first-out under overload serves the freshest, most likely live requests. This matters for exactly the stale-queue failure that destroyed goodput in the simulated naive run.

Figure 4: A layered resilience stack for outbound calls. Each control addresses a different failure: deadlines stop stale work, bulkheads isolate pools, breakers stop traffic to dead dependencies, and the retry budget bounds extra load.

Figure 4 shows the order in which these controls are usually composed on an outbound call. The deadline check comes first so doomed requests never consume a permit. The retry loop wraps everything below it, so every attempt is judged independently by the bulkhead and the breaker; a rejected attempt is cheap and consumes no budget beyond its token. Each dependency gets its own bulkhead and breaker pair, and the upstream service ends the chain with its own load shedder. Nesting order changes semantics, so check your library: Resilience4j’s Spring Boot integration documents its own default aspect order, with retry outermost and bulkhead innermost, which differs from the bulkhead-then-breaker order drawn here.

Idempotency, Simulation Code and the Library Landscape

Idempotency is a precondition, not a feature

A retry is only safe if executing the operation twice has the same effect as once. Reads and PUT-style replacements are naturally idempotent. A POST that charges a card is not. The failure mode is the lost response: the server charged the card, the reply was dropped, the client retried, and the customer was charged twice.

The standard fix is an idempotency key. The client generates a unique key per logical operation and sends it on every attempt; the server stores the key with the result and returns the stored result for repeats. Two details decide whether it works. The key must be recorded atomically with the side effect, in the same transaction, or a crash between them re-opens the window. And the stored result needs a retention period longer than the longest plausible retry horizon. Message-streaming systems face the same issue; see our deep-dive on Kafka exactly-once semantics, transactions and idempotent producers for the broker-side version.

The simulation, in full

This is the script behind the table earlier. It models a FIFO backend and clients that abandon requests after one second. Run it with python3 sim_retry_storm.py; every number is illustrative and depends on the constants at the top.

from collections import deque

CAPACITY, BASE_RPS = 1000, 800
TRIGGER = (20, 25)              # capacity falls to 40 percent for 5 seconds
TIMEOUT, MAX_ATTEMPTS = 1, 4

def run(policy, seconds=90):
    q = deque()                 # items: [arrival_second, attempt, stale]
    tokens, open_until, log = 0.0, -1, []
    for t in range(seconds):
        cap = int(CAPACITY * 0.4) if TRIGGER[0] <= t < TRIGGER[1] else CAPACITY
        retries = 0
        for item in list(q):
            arr, att, stale = item
            if not stale and t - arr >= TIMEOUT:
                item[2] = True                      # client gave up on this copy
                allowed = att < MAX_ATTEMPTS and policy != "none"
                if allowed and policy != "naive":
                    breaker_open = policy == "budget+breaker" and t < open_until
                    if tokens >= 1 and not breaker_open:
                        tokens -= 1
                    else:
                        allowed = False
                if allowed:
                    q.append([t, att + 1, False]); retries += 1
        for _ in range(BASE_RPS):
            q.append([t, 1, False])
        good = 0
        for _ in range(min(cap, len(q))):
            arr, att, stale = q.popleft()
            if not stale and t - arr < TIMEOUT:
                good += 1
        tokens = min(100.0, tokens + 0.1 * good)    # 10 percent of good replies earn a token
        offered = BASE_RPS + retries
        if policy == "budget+breaker" and good < 0.5 * offered:
            open_until = t + 5
        log.append((t, offered, good, len(q)))
    return log

Three things are worth noticing. During the five-second trigger the naive policy keeps three or four copies of every request alive, so the queue grows faster than the recovered backend can drain it. After the trigger passes, the backend serves requests that are already stale, so goodput stays at zero: the system is working at full capacity and delivering nothing. And in the budgeted runs the retry rate can never exceed 10% of good replies plus the token cap, so the queue stays empty. The breaker adds little here because the budget already prevents the damage; in a real system its value shows when a dependency is hard-down and every call would otherwise burn a thread for a full timeout.

Notice also what the simulation leaves out. It has no jitter, because arrivals are uniform per second, and no load-dependent failure curve beyond the queue. Real systems have richer dynamics, so use this as a mechanism demonstration and measure your own with a load test.

A Go budget you can embed

A retry budget is a few lines. This Go version uses the gRPC-style token bucket: each failure costs a token, each success earns a fraction, and retries are allowed above half the maximum.

type Budget struct {
    mu     sync.Mutex
    tokens float64
    max    float64
    ratio  float64
}

func NewBudget(max, ratio float64) *Budget { return &Budget{tokens: max, max: max, ratio: ratio} }

func (b *Budget) OnSuccess() { b.mu.Lock(); b.tokens = math.Min(b.max, b.tokens+b.ratio); b.mu.Unlock() }
func (b *Budget) OnFailure() { b.mu.Lock(); b.tokens = math.Max(0, b.tokens-1); b.mu.Unlock() }
func (b *Budget) AllowRetry() bool {
    b.mu.Lock(); defer b.mu.Unlock()
    return b.tokens > b.max/2
}

The library landscape

You rarely write these from scratch. Resilience4j is the maintained Java library with circuit breaker, retry, bulkhead, rate limiter and time limiter modules, and a Spring Boot integration; it replaced Netflix Hystrix, which is in maintenance mode. Polly (now documented as resilience strategies, with retry, circuit breaker, timeout, hedging and rate limiter) is the standard in .NET. In Go, teams commonly use small libraries such as sony/gobreaker for breakers and rely on gRPC’s built-in retry and hedging configuration. In Python, tenacity handles retries and backoff.

At the infrastructure layer, Envoy and the service meshes built on it (Istio among them) move these controls out of application code: retry policies on routes, RetryBudget and circuit-breaker thresholds on clusters, and outlier detection for host ejection. The trade-off is operational visibility. Mesh-level retries are invisible to application developers, so a mesh retry of three plus an application retry of three is nine, the same multiplication from Figure 1 with a different owner. Decide which layer owns retries for each call path and disable the others.

Concern	Application library	Service mesh or proxy
Per-operation idempotency knowledge	Yes, knows which calls are safe	No, must be told via route config
Uniform policy across languages	No, per-language	Yes
Business-aware fallbacks	Yes	No
Visibility to developers	High	Low unless metrics are exposed
Risk of double retry layers	Medium	High if app also retries

Trade-offs, Gotchas, and What Goes Wrong

Every protection in this article can itself cause an outage. The common failure modes fall into a few families.

Synchronized probes and herds. A fleet of 500 instances with identical breaker settings opens at the same moment and, 60 seconds later, probes at the same moment. Even with a small trial-call quota per instance, the dependency sees 500 times the quota. Randomize wait durations, and treat half-open as a fleet-level concern.

Breakers that mask real errors. An open breaker returns an error that looks like a dependency failure. If dashboards count it as one, you lose the ability to tell whether the dependency is still down or the breaker is merely still open. Emit distinct metrics for rejected-by-breaker, rejected-by-bulkhead and rejected-by-budget, and alert on each.

Budgets that are too tight. A budget of 10% protects against storms but also suppresses retries during an ordinary regional blip where retrying would have worked. The fix is the floor in Envoy’s min_retry_concurrency and a sensible token cap, and separate budgets per dependency so one noisy service does not drain the budget for the others.

Thresholds tuned on quiet traffic. A breaker with minimumNumberOfCalls of 100 never opens on an endpoint that sees 3 calls per minute, and one tuned on peak traffic flaps at night. Test both ends of your traffic range.

Mismatched timeouts. If the client timeout is shorter than the server’s processing time, every slow request is a client failure plus a completed server action. Make the outer timeout longer than inner ones, and propagate deadlines so the inner ones shrink automatically.

Retrying non-idempotent writes. Covered above, but it deserves repeating as the highest-severity mistake. Disable automatic retries on writes that lack idempotency keys, even if the library defaults say otherwise.

Fallbacks that fail more. A fallback to a cache or a default response is only safe if the fallback path is itself capacity-tested. A fallback that triggers a heavier computation, or hits a different, smaller datastore, converts a partial outage into a total one. Chaos experiments are the way to find out; our guide to chaos engineering on Kubernetes with Chaos Mesh and Litmus shows how to inject latency and error faults against a defined steady state.

Layer duplication. Retries in the client SDK, the mesh sidecar, the gateway and the application multiply. The answer is ownership: one layer retries, the rest pass errors through.

The honest summary is that these controls trade peak success rate for survival. In a healthy system a budgeted retry policy occasionally returns an error a free-for-all retry would have turned into a success. That cost is real and small, and it is the price of a bounded worst case.

Practical Recommendations

Start from the dependency graph rather than from the libraries. For each outbound call, decide who owns retry, what the deadline is, whether the operation is idempotent, and what the caller does when the answer is no. Then apply controls in order of leverage. First, deadlines and the single-owner retry rule; these cost nothing and remove the multiplicative chain. Second, a retry budget with jittered backoff. Third, per-dependency bulkheads. Fourth, circuit breakers with slow-call detection. Fifth, hedging where tail latency justifies it and the calls are idempotent.

Tune from measurements, not from defaults. Pull the real latency distribution of each dependency, set the slow-call threshold just above the 99th percentile you are willing to tolerate, set the breaker window to span a few seconds of typical traffic, and set the per-request attempt cap at 3 or fewer. Then prove it: inject a latency fault and a hard-down fault in a staging environment with production-like load, and watch the offered-load curve. If it rises by more than your budget percentage, something is retrying that you did not know about.

Checklist:

[ ] Exactly one layer retries each call path; all others pass errors through.
[ ] Every retried operation is idempotent or carries an idempotency key.
[ ] Backoff is capped and uses full or decorrelated jitter.
[ ] A retry budget bounds fleet-wide retries, with a minimum floor.
[ ] Per-request attempts are capped at 3 or fewer, and total retry time is inside the deadline.
[ ] Deadlines propagate and shrink at every hop.
[ ] Each dependency has its own bulkhead and breaker, with slow-call detection enabled.
[ ] Half-open admits a few probes, requires several successes and uses jittered wait times.
[ ] Servers shed load fast and return a “do not retry” signal when overloaded.
[ ] Rejections by breaker, bulkhead and budget are separate metrics with alerts.
[ ] A fault-injection test shows offered load staying within the budget under a hard failure.

Frequently Asked Questions

What is the circuit breaker pattern in microservices?

The circuit breaker pattern wraps a remote call in a state machine with closed, open and half-open states. While closed, calls pass and outcomes are counted. When the failure or slow-call rate crosses a threshold, the breaker opens and rejects calls instantly. After a wait it lets a few trial calls through; success closes it again and failure reopens it. The goal is to stop adding load to a failing dependency and to fail fast.

What is the difference between a retry and a circuit breaker?

A retry reacts to one failed call by trying again, which helps with brief transient faults. A circuit breaker reacts to the pattern of many failures by stopping calls entirely for a period. They solve opposite risks: retries add load, breakers remove it. Used together, the breaker should sit inside the retry loop so each attempt is checked, and a rejected attempt should not trigger further retries or consume a retry token.

Why does exponential backoff need jitter?

Without jitter, clients that failed at the same instant all wait the same exponential delays and retry in synchronized waves, each as tall as the original burst. Jitter randomizes the wait so retries spread in time. The AWS Architecture Blog analysis found that jittered variants reduce total work substantially compared with no jitter. Full jitter, a uniform random wait between zero and the capped exponential value, is a sound default.

What is a retry budget and how big should it be?

A retry budget caps retries as a fraction of normal traffic, so total extra load stays bounded. Google’s SRE book gives 10% per client as an example, and Envoy’s RetryBudget defaults to 20% with a minimum concurrency of 3. The right value depends on how often your dependency fails transiently. Start near 10%, measure how often legitimate retries are refused, and raise it only if success rates during blips suffer.

How do bulkheads differ from circuit breakers?

A bulkhead limits how much of a shared resource, such as threads or connections, one dependency may use, so a slow dependency cannot starve the others. A circuit breaker decides whether to call a dependency at all, based on its recent health. Bulkheads work before you know anything is wrong, since they cap concurrency from the first call. Breakers need a window of observed failures before they act.

When should I use hedged requests?

Use hedging for idempotent reads with a heavy latency tail, where a second copy to another replica can beat a slow one. The Tail at Scale paper reported cutting 99.9th-percentile latency from 1,800 ms to 74 ms with 2% extra requests in a BigTable benchmark. Do not hedge against overload, because extra copies add pressure, and keep hedges inside the same throttling budget as retries so they shut off when the cluster is unhealthy.