Overload Control: Backpressure, Load Shedding, Adaptive Concurrency Limits and CoDel (2026)

A service that is 20 percent over capacity does not serve 80 percent of its traffic. Left alone, it frequently serves close to none of it. Requests pile up in queues, every response arrives after the client has given up, the client retries, and the retries add more work than the original traffic did. The server is busy at 100 percent CPU and produces almost no useful output. This is the signature of an overloaded system, and it is why load shedding is not a nice-to-have fallback but the central mechanism that keeps a service alive when demand exceeds supply.

This matters more in 2026 than it did a decade ago, because deep microservice call graphs, aggressive client retry libraries, and autoscalers that react in minutes rather than seconds make overload both more likely and harder to escape. This post builds the full overload-control stack from first principles: queue math with Little’s Law, the difference between throughput and goodput, backpressure, shedding policies, Netflix-style adaptive concurrency limits, CoDel, and client-side throttling. You will leave with a numeric worked example, a decision framework, and a list of failure modes to test before production does it for you.

What this covers: why overload collapses goodput, how metastable failures sustain themselves, the four layers of defence (client, edge, server queue, admission), and how to tune each with real numbers.

Context and Background

Overload control has three intellectual roots. The first is queueing theory, which tells us that a queue in front of a saturated server grows without bound and that waiting time, not service time, dominates latency. The second is network congestion control: TCP’s additive-increase multiplicative-decrease (AIMD) loop, the Vegas delay-based variant, and active queue management such as CoDel all solve the same problem of finding the right amount of in-flight work without being told the capacity. The third is operational experience: the Google SRE book’s chapters on handling overload and addressing cascading failures, and the Amazon Builders’ Library article on using load shedding to avoid overload, which distilled a decade of incident reviews into policy.

The vocabulary matters because teams use these terms loosely. Backpressure is a signal flowing upstream that says “slow down”, delivered by blocking, by explicit credits, or by a refusal. Load shedding is the server deliberately rejecting or dropping work it has decided it cannot finish in time. Admission control is the broader decision of whether to let a request in at all, using cost, priority and tenant. Rate limiting caps a caller’s rate against a fixed policy and is covered in detail in our guide to distributed rate limiting with token bucket, GCRA, Redis and Envoy. Rate limits protect a service from an individual caller; overload control protects it from the aggregate, including the aggregate of well-behaved callers whose combined demand exceeds capacity today.

The incumbents are well known. Netflix open-sourced the concurrency-limits library, which implements Vegas- and gradient-style limiters for gRPC and servlet stacks. Envoy ships an adaptive concurrency HTTP filter and an overload manager that ties resource pressure to actions such as refusing new requests. Google describes client-side adaptive throttling and criticality-based shedding in its SRE book chapter on handling overload. Linux has shipped CoDel as a queue discipline for years, and the IETF standardised it as RFC 8289 (an Experimental-status document).

The reason this stack matters is the failure class it addresses. Bronson, Aghayev, Charapko and Zhu introduced metastable failures at HotOS 2021: systems that, after a temporary trigger, remain in a bad state because a sustaining feedback loop keeps load at or above capacity even once the trigger is gone. A follow-up OSDI 2022 study by Huang and colleagues, “Metastable Failures in the Wild”, examined public incident reports and reported that retries were the sustaining effect in more than half of the incidents, and that load shedding was the most common mitigation. That second point is telling: operators reach for load shedding after the fact, under pressure, with no clear sense of how much to shed. The rest of this post is about designing it before you need it.

The Overload Control Stack: A Reference Architecture

Direct answer: Overload control is a layered defence. Clients throttle themselves and budget retries; the edge sheds by priority; each server bounds its concurrency with an adaptive limit and its queue with a delay target; and expensive work is admitted only if its deadline can still be met. No single layer is sufficient, because each one fails against a different overload shape.

Figure 1: Four layers of overload control. Each layer converts excess demand into a cheap, early refusal rather than an expensive, late timeout.

Figure 1 shows the reference design. A request leaves a client that holds a retry budget and an adaptive throttle. It passes an edge or proxy tier that performs coarse load shedding by criticality and tenant quota. At the server, a concurrency limiter decides how many requests may be in flight, a bounded queue with a delay target (CoDel or a hard age limit) holds the overflow, and a deadline check at dequeue time discards work whose caller has already given up. The refusal path is deliberately cheap: it must cost far less CPU than doing the work, or shedding itself becomes the overload.

Why queues are the root of the problem

Little’s Law states that the average number of items in a stable system equals arrival rate times average time in system: L = λW. It holds regardless of distribution, which is what makes it useful. Consider a server with 100 worker threads and a mean service time of 50 ms. Its capacity is 100 / 0.05 = 2,000 requests per second. At 1,900 rps the system is stable with a small queue. At 2,400 rps, the surplus of 400 rps accumulates in the queue. The queue never drains, and the wait time rises linearly: after t seconds there are 400t queued requests, and a newly arrived request waits 400t / 2,000 = 0.2t seconds.

If clients time out after one second, a request arriving once the queue holds 2,000 items (t = 5 s) will wait a full second and be abandoned before service begins. From that moment, under FIFO scheduling, every request the server dequeues is one whose caller has already left. The server is working at full speed on results nobody reads. Throughput is still 2,000 rps; goodput is approximately zero. The numbers above are illustrative, but the structure is exact: the collapse arrives after queue wait crosses the client timeout, and it arrives in seconds, not minutes.

Throughput versus goodput

Throughput is the rate at which a server completes requests. Goodput is the rate at which it completes requests that the caller can still use. The Amazon Builders’ Library frames the objective as keeping goodput flat as offered load rises past capacity, instead of falling off a cliff. A healthy overload design produces a goodput curve that rises with load, plateaus at capacity, and stays on the plateau. An unprotected system produces a curve that rises, peaks, and then declines because work wasted on timed-out requests grows with load.

Three mechanisms drive the decline, and a design should name which one it addresses. Queue wait exceeding client timeouts wastes the work of serving stale requests. Contention (lock convoys, cache thrashing, garbage collection, context switching) raises the per-request cost as concurrency grows beyond the knee, so capacity itself shrinks under load. And retries multiply the offered load. Concurrency limits attack the second, queue management attacks the first, and retry budgets attack the third.

Backpressure: pushing the signal upstream

Backpressure is the honest answer to “what happens when I cannot keep up?” A bounded queue that blocks its producer is the simplest form. In streaming systems it is explicit: Reactive Streams uses demand signalling via request(n), HTTP/2 and gRPC use per-stream and per-connection flow-control windows, and TCP uses the receiver window. In request-response systems it is implicit and cruder: an HTTP 503 or 429, a gRPC UNAVAILABLE or RESOURCE_EXHAUSTED, or a refused connection.

The design rule is that backpressure must reach a component that can actually reduce demand. Blocking a thread in the middle of a call chain merely moves the queue to a different place, often one with less memory or a less visible metric. Backpressure is complete only when it reaches the original source: a user who sees a “try again shortly” page, a batch job that pauses, or a sensor gateway that buffers locally and backs off. In IoT ingestion this is the difference between a broker that applies per-connection flow control and one that accepts everything into an unbounded queue until the heap fills.

Load shedding: choosing what to drop

Backpressure asks the sender to slow down; load shedding takes the decision away from the sender. When arrival exceeds capacity, something must be refused. The only question is whether the refusal is chosen or accidental. Accidental shedding looks like out-of-memory kills, thread pool exhaustion, and timeouts in random places. Chosen shedding is a policy.

Common policies, ordered roughly by sophistication, are these. Random or probabilistic shedding rejects a fixed fraction and is simple, but it treats a payment and a prefetch identically. Priority shedding sorts requests by criticality; Google’s SRE book describes four levels (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE) propagated across RPC hops so a backend can shed the lowest class first. Tenant-aware shedding applies per-customer quotas, so one noisy caller cannot consume the shared pool. Deadline-aware shedding refuses work whose remaining time is shorter than the expected service time. Cost-aware shedding refuses expensive operations first, since rejecting one heavy query saves as much capacity as rejecting twenty light ones.

Deeper Analysis: Mechanisms, Numbers, and Algorithms

Metastable failure and the retry storm

A metastable failure has three parts: a vulnerability (the system runs with load close enough to capacity that a perturbation can tip it), a trigger (a load spike, a deployment, a cache flush, a failover), and a sustaining effect (a loop that keeps overload alive after the trigger ends). The OSDI 2022 study defines the bad state as permanent overload with ultra-low goodput. Its data set is public postmortems, and the authors report that retries were the sustaining effect in over half of the incidents, with triggers split between load spikes and engineer errors such as bad configuration or code.

Figure 2: The retry feedback loop. A trigger pushes latency over the client timeout; timeouts create retries; retries raise offered load; the loop sustains itself after the trigger is gone. Shedding and retry budgets are the two cut points.

The loop in Figure 2 is easy to quantify. Suppose a service has capacity C = 2,000 rps and a steady base demand of 1,000 rps, comfortably at 50 percent utilisation. A deploy briefly cuts capacity, latency crosses the client timeout, and clients begin retrying failed calls up to three times. If every attempt fails, the offered load becomes 1,000 × 4 = 4,000 rps, twice capacity. Now restore full capacity: the system still faces 4,000 rps while it can serve 2,000, the queue grows, latency stays above the timeout, and failures keep generating retries. The trigger is gone and the outage persists. This is a stable bad equilibrium, and no amount of waiting fixes it.

The Google SRE book gives the multi-layer version. If each of three layers retries up to four attempts, a single user request can become 4³ = 64 attempts at the database. Their retry guidance is concrete: a per-request cap of three attempts, and a per-client retry budget that allows retries only while they are under roughly 10 percent of request volume. With that budget, worst-case amplification is bounded near 1.1x instead of near 3x, according to the SRE book. A budget changes the shape of the problem: it removes the positive feedback and with it the second stable state.

Two further properties make these failures nasty. First, the stable threshold falls: the study notes that with no cap on retries the system has no stable region at all. Second, fixing the trigger is insufficient, so the only exits are to shed load until the system drops below its stable threshold, or to raise capacity. Operators often lack a measurement of how far to shed and therefore reboot servers, which clears queues but also caches, and can re-trigger the failure when traffic returns to a cold fleet.

Shedding policies in detail

Queue discipline: FIFO versus adaptive LIFO. FIFO is fair in the calm regime and catastrophic in the overloaded one, because it serves the oldest requests first, which are the ones most likely to have expired. LIFO serves the newest first, which are the ones most likely to still have a waiting caller. The Amazon article recommends LIFO where the protocol permits it, and the SRE book lists LIFO and CoDel as options for dropping requests unlikely to be worth processing. The mature form is adaptive LIFO: run FIFO while the queue is short for fairness, and switch to LIFO when the queue depth or head-of-line age passes a threshold. LIFO has a price. The unlucky old requests starve indefinitely, so pair it with a maximum age after which they are discarded, and accept that tail latency for the starved class becomes a timeout.

Deadline propagation. Every request should carry an absolute or remaining-time deadline across hops. The SRE book gives the arithmetic: a server that has consumed 7 seconds of a 30-second deadline passes 23 seconds downstream, and checks remaining time before each expensive stage. The Builders’ Library cautions on implementation: absolute timestamps need synchronised clocks, while durations need a monotonic timer and a good estimate of when the clock started, which is hard when requests sit in kernel buffers. Whichever you pick, enforce the check at dequeue time. A request that waited 900 ms in a queue against a 1 s deadline should be dropped immediately if the service time is 50 ms or more, rather than started and abandoned mid-flight.

Priority and criticality. Priority only helps if it is assigned by the business meaning of the call, not by the caller’s self-interest. In practice, internal batch traffic and speculative prefetch go to the lowest class; user-blocking reads and writes go to the default; a very small set of calls (health checks from load balancers, in-progress transaction completion) get the highest class. The Builders’ Library makes the last point specifically: always prioritise load balancer ping requests, because failing them removes capacity, and prioritise calls that finish in-flight work over ones that start new work.

Admission control by cost. For services with heterogeneous requests, the number of requests is a poor unit. A better unit is estimated cost: CPU-seconds, rows scanned, or tokens generated. Google’s per-customer quota example is expressed in CPU-seconds per second, with quotas allowed to sum to more than physical capacity because customers rarely peak together. Token-count budgeting for LLM inference follows the same logic: one 100,000-token request is not equivalent to one 100-token request.

Adaptive concurrency limits

A fixed concurrency cap is the simplest shedder: allow N requests in flight, refuse the rest. The trouble is choosing N. Too high and queues form inside the server; too low and capacity is wasted. The right N moves with request mix, downstream health, instance size and noisy neighbours. Adaptive concurrency limits solve this by borrowing from TCP congestion control: continuously probe for the largest in-flight count that does not inflate latency.

Figure 3: An adaptive concurrency limiter. Observed round-trip time against a tracked minimum drives the limit up or down; excess requests beyond the limit are rejected immediately.

Netflix’s concurrency-limits library implements several algorithms. A pure loss-based AIMD limit increases additively on success and cuts multiplicatively when a request is dropped or times out, exactly as TCP Reno does. Vegas is delay-based: it estimates the queue at the bottleneck as limit × (1 − minRTT / sampleRTT) and, at the end of each sampling window, raises the limit by one if the estimated queue is below a lower threshold alpha (typically 2 to 3) and lowers it by one if the queue is above an upper threshold beta (typically 4 to 6). Gradient2 compares a long-window and a short-window exponential average of latency, treating sustained divergence as evidence of queue build-up, and is designed to reduce the bias and drift that minimum-latency tracking introduces.

A worked Vegas step makes this concrete (all numbers illustrative). Suppose limit L = 100, the best latency ever seen is minRTT = 10 ms, and this window’s sample latency is 20 ms. The estimated queue is 100 × (1 − 10/20) = 50. That is far above beta, so the limit steps down to 99, and keeps stepping down each window while latency stays inflated. If instead sample latency is 10.4 ms, the queue estimate is about 3.8, which sits between alpha and beta, and the limit holds. The additive steps make convergence gentle, which is good for stability and bad for reacting to sudden capacity loss; many deployments therefore cut multiplicatively when requests are dropped, combining loss and delay signals.

Envoy’s adaptive concurrency HTTP filter uses a gradient controller whose documented update is gradient = (minRTT + B) / sampleRTT, where B = minRTT × buffer percentage, and limit_new = gradient × limit_old + headroom, with headroom fixed at the square root of the limit. Take minRTT = 10 ms, a 10 percent buffer (B = 1 ms), a sampled p90 latency of 20 ms and a current limit of 100. The gradient is 11 / 20 = 0.55; the new limit is 0.55 × 100 + √100 = 55 + 10 = 65. If latency returns to 11 ms, the gradient becomes 1.0 and the limit grows by about the headroom each update: 100 + 10 = 110. The headroom term is what lets the controller keep probing for more capacity while the gradient term pulls the limit down fast when latency rises. Envoy measures minRTT periodically by pinning the limit to a small value (a default of 3 during measurement) and re-measures if the limit has sat at its floor for five consecutive windows, with jitter so a fleet does not re-measure in lockstep.

CoDel: controlling delay, not length

Kathleen Nichols and Van Jacobson’s CoDel (Controlled Delay) was designed for network buffers, but its idea transfers directly to request queues. The insight is that queue length is the wrong signal: a long queue that drains in a burst is harmless, while a short queue that never empties is a standing queue that adds latency with no benefit. CoDel measures sojourn time, the time each item spends queued, and tracks the minimum sojourn time over a sliding interval. If even the minimum stays above a target, the queue is persistently too deep rather than merely bursty.

RFC 8289 gives defaults: TARGET of 5 ms and INTERVAL of 100 ms for normal Internet traffic, where INTERVAL should roughly cover a worst-case round-trip time. CoDel enters its dropping state only after sojourn time has stayed above the target for a full interval, drops one item, and schedules the next drop at t + INTERVAL / √count, where count is the number of drops since entering that state. The control law tightens as pressure persists: the spacing between drops is 100 ms, then 70.7 ms, 57.7 ms, 50 ms and 44.7 ms for counts one through five. When sojourn time falls back under the target the state is left, and a non-starvation rule means CoDel will not drop when less than one MTU of data would remain.

Figure 4: CoDel in a request server. Each request is timestamped on enqueue; at dequeue the sojourn time is compared with the target, and persistent excess triggers drops spaced by interval over the square root of the drop count.

For services, 5 ms and 100 ms are network numbers and must be rescaled. A practical translation sets INTERVAL to a few times the typical service time or the p99 of a healthy request, and TARGET to a small fraction of it, so a standing queue is detected quickly. A service with a 50 ms mean service time might use a target near 5 to 10 ms of queue wait and an interval of a few hundred milliseconds; this is a starting point for tuning, not a published recommendation. The benefit over a hard queue-length cap is that CoDel adapts to the service rate: when the server is fast the same queue length drains quickly and is left alone, and when it is slow the identical depth is a problem and is policed. The SRE book names CoDel, together with LIFO, as an option and notes it works best combined with deadline propagation.

Putting the numbers together

Return to the 2,000 rps server and apply the stack to an offered load of 2,400 rps with a one-second client timeout. All figures are illustrative arithmetic, not measurements.

Design	Queue behaviour	Goodput at 2,400 rps offered	Failure mode
Unbounded FIFO queue	Wait exceeds 1 s after about 5 s	Near zero once the queue exceeds 2,000 items	Metastable; retries add more load
Bounded queue of 200 items	Maximum wait about 0.1 s	Close to 2,000 rps; 400 rps get fast 503s	Fixed bound mis-sized if service time changes
Adaptive concurrency limit	In-flight pinned near the knee	Close to capacity; excess rejected at entry	Slow reaction to sudden capacity loss
Limit plus CoDel plus LIFO	Standing queue trimmed; newest served	Close to capacity with lowest tail latency	Starved old requests; needs parameter care

A queue bound of 200 comes from Little’s Law in reverse: we want a maximum wait of 100 ms at a service rate of 2,000 rps, so L = λW = 2,000 × 0.1 = 200. The same formula sets the bound for any latency budget. A service that must answer within 300 ms when service time is 50 ms can afford a queue wait of about 250 ms at most, so a queue of roughly 500 items at 2,000 rps, and should shed beyond it. The point of every row after the first is that excess demand becomes 400 rps of cheap rejections instead of 2,400 rps of expensive timeouts.

Client-side throttling and retry budgets

The server cannot be the only line of defence, because rejecting a request still costs a connection, a parse and a response. If rejection is cheap but not free, a large enough flood of rejected requests still saturates the front door. Google’s answer is adaptive client-side throttling. Each client counts, over the last two minutes, the requests it attempted and the requests the backend accepted. When requests exceed K times accepts, the client starts rejecting new requests locally with a probability that grows with the gap. The SRE book gives the formula max(0, (requests − K × accepts) / (requests + 1)), with a default K of 2; setting K nearer 1.1 throttles more aggressively. With K = 2, a client that sees 1,000 attempts and 400 accepts rejects about (1,000 − 800) / 1,001 ≈ 20 percent of new calls locally. If accepts fall to 250, the figure is about 50 percent. The behaviour is smooth, needs no coordination, and is self-correcting, because when the backend recovers the accepts count catches up and throttling relaxes.

This pairs with a retry budget. The budget limits retries to a fraction of normal traffic, and the throttle drops unsent attempts when the backend is visibly refusing. Add jittered exponential backoff so that retries from many clients do not align, and make retries idempotent by design. The most important property is that retries happen at one layer. If the edge, the service mesh and the SDK each retry three times, the multiplication is the 64x outcome described above.

Resource-triggered shedding in the proxy: Envoy overload manager

Concurrency limits react to latency. Sometimes the right trigger is a direct resource signal, such as heap usage, CPU, cgroup memory or connection count. Envoy’s overload manager takes resource monitors (fixed heap, CPU utilisation, cgroup memory, global downstream connections) and maps their pressure onto actions through triggers. A threshold trigger is binary; a scaled trigger ramps linearly between a scaling threshold and a saturation threshold. Actions include stopping acceptance of new requests (new requests get a 503), disabling HTTP keep-alive, refusing new connections, shrinking the heap, and scaling down timeouts.

The scaled form is worth copying elsewhere in your own designs, because it avoids a cliff. Envoy’s documentation gives an example: a 600 s idle timeout scaled down to a 2 s minimum between 85 and 95 percent heap pressure yields about 181 s at 92 percent. The principle is gradual degradation in proportion to pressure rather than a single switch that flips traffic from fully served to fully refused.

Trade-offs, Gotchas, and What Goes Wrong

Overload control creates its own failure modes, and most are discovered during incidents. They are worth walking through deliberately.

Shedding that costs as much as serving. If the reject path parses the body, authenticates, logs verbosely and writes a metric before refusing, a flood of rejects can use more CPU than the work it avoided. The Builders’ Library notes that shedding is not free and that goodput eventually falls anyway, an Amdahl’s-law effect. Reject as early and as cheaply as possible, ideally before deserialisation, and sample the logs.

Shedding that hides the problem. Rejections count as fast responses, so median latency can look excellent while a large share of users get errors. The Builders’ Library gives the example of a service shedding 60 percent of traffic with a flattering median. Alert on goodput, rejection rate and latency of successful requests, not on overall latency. The same article warns that shedding can disable reactive autoscaling if both are triggered by a similar CPU target, because a shed service never looks busy enough to scale.

Concurrency limits fooled by the wrong baseline. Delay-based limiters assume that the minimum observed latency represents an unloaded system. If the first minutes after a deploy are slow because of a cold cache or JIT warm-up, minRTT is overestimated and the limiter tolerates queueing. If a lucky fast sample sets minRTT too low, the limiter permanently over-throttles. Envoy’s periodic re-measurement with jitter and Gradient2’s long-versus-short windows exist precisely to mitigate drift. Variable-cost endpoints are another trap: one limiter across cheap reads and expensive reports will be tuned by whichever dominates, so partition by request class.

Partition starvation and priority inversion. Netflix’s library supports percentage partitions, for example reserving 90 percent of the limit for live traffic and 10 percent for batch, with unlisted groups allowed only excess capacity. Good, but a batch partition sized too small can starve a job that others depend on. If a high-priority request waits on a lower-priority downstream call, the priority is lost at the hop; propagate it.

LIFO and CoDel surprises. LIFO makes the oldest requests wait forever, so without a maximum-age discard the queue accumulates zombies. CoDel’s dropped item may not be the one that matters, and the control law that suits a single TCP flow is not guaranteed to suit many independent bursty request streams, as commenters on the ACM Queue paper pointed out (Nichols herself noted that no control law is perfect). Treat the published constants as starting points.

Degraded modes that were never exercised. The SRE book warns that rarely used degradation paths fail when needed, and recommends running a small set of servers near overload regularly to test them. Add a kill switch: complex overload mechanisms can trip unintentionally.

Cold starts after recovery. When load shedding ends, traffic returns to servers with cold caches and un-warmed JIT. Ramp the admitted fraction up slowly instead of releasing everything at once, or the system re-enters overload at the moment it is most fragile.

Practical Recommendations

Start with measurements, not mechanisms. Find the knee of your service: ramp synthetic load in a staging environment and plot goodput, p99 latency and CPU against concurrency. The knee, where latency starts rising faster than throughput, tells you the concurrency to target and shows whether the service even has a plateau. Many services do not, because contention makes goodput fall beyond the knee, and for them an early, firm limit matters most.

Then build outward from the cheapest, most general protections. Put a hard cap on queue age and size so no request can wait longer than its caller will. Add deadline propagation and check the deadline at dequeue. Introduce an adaptive concurrency limit per service, partitioned by request class, with a floor that never starves health checks. Add CoDel or an equivalent delay target if queues are deep for legitimate reasons. Set retry budgets and adaptive throttling in the client library so every team gets them by default, and assign criticality at the edge.

Operate it like a safety system. Alert on goodput and shed rate. Rehearse overload by running game days that push a canary shard to 120 percent of capacity. Make shedding changes behind flags with fast rollback. If you run cell-based architecture, shed at the cell boundary so a hot cell does not drag the others down; our piece on cell-based architecture, bulkheads and blast radius covers how isolation limits the reach of an overload.

A short checklist:

• Measure the goodput knee and set a concurrency target below it.
• Bound every queue by size and by age; derive the bound from Little’s Law and the latency budget.
• Propagate deadlines and enforce them at dequeue time.
• Retry at one layer only, with a per-request cap and a per-client budget near 10 percent.
• Assign criticality classes and protect health checks and in-flight completions.
• Make rejection cheap, early and observable; alert on goodput, not average latency.
• Test the degraded modes and the recovery ramp before production does.

Frequently Asked Questions

What is the difference between backpressure and load shedding?

Backpressure asks the sender to slow down, by blocking, by flow-control credits or by a refusal it is expected to honour. Load shedding is the receiver unilaterally discarding work it cannot finish in time. Backpressure keeps work in the system, which is efficient when producers cooperate. Load shedding protects the receiver when they do not, or when the producers are external users. Robust designs use both: backpressure within trusted pipelines, shedding at trust and capacity boundaries.

How does load shedding improve availability if it rejects requests?

An overloaded system without shedding tends to serve nearly everyone badly, because queue wait exceeds client timeouts and goodput collapses. By refusing the excess early, the server finishes the requests it accepted within their deadlines. In the worked example above, a 2,000 rps server at 2,400 rps offered load rejects 400 rps but serves about 2,000 successfully, versus serving close to none under an unbounded FIFO queue. Controlled partial failure beats uncontrolled total failure.

What are adaptive concurrency limits and when should I use them?

Adaptive concurrency limits automatically adjust the number of in-flight requests a service allows, using signals such as latency growth (Vegas, Gradient) or drops (AIMD), borrowed from TCP congestion control. Use them when capacity varies with request mix, instance type or downstream health, so a static number would be wrong much of the time. They are a poor fit for services with very low request volume, where there are too few samples to estimate latency reliably.

Is CoDel only for network routers?

No. CoDel was specified for network buffers in RFC 8289, but its principle applies to any queue: police the persistent minimum delay, not the length. The Google SRE book names CoDel as a way to drop requests unlikely to be worth processing. The published constants of 5 ms and 100 ms are tuned for Internet round-trip times, so a service must rescale them to its own service-time distribution and validate under load.

Why do retries make overload worse?

Each failed or timed-out request produces additional attempts, so offered load rises exactly when capacity is scarce. With three retries per request, a fully failing client generates four times the traffic, and across several service layers the multiplication compounds, as in the SRE book’s 4³ = 64 example. The OSDI 2022 metastable-failures study found retries were the sustaining effect in over half of the incidents it examined. Retry budgets, single-layer retries and jittered backoff break the loop.

What does the Netflix concurrency-limits library do?

It is a Java library that measures request latency and drops and adjusts a concurrency limit using algorithms including AIMD, Vegas and Gradient2, then rejects requests that exceed it. It integrates with gRPC and servlet filters, can partition the limit by request group so batch cannot starve live traffic, and returns an unavailable or HTTP 429-style response to rejected callers. The repository documents the algorithms and the integration examples.

Further Reading

• Distributed rate limiting architecture: token bucket, GCRA, Redis and Envoy for per-caller limits that complement aggregate overload control.
• Cell-based architecture, bulkheads and blast radius: a reference design for confining an overload to one cell.
• Handling Overload and Addressing Cascading Failures, Google SRE book for criticality levels, client-side throttling and retry budgets.
• Metastable Failures in the Wild, OSDI 2022 and RFC 8289, Controlled Delay Active Queue Management for the primary sources on sustaining effects and CoDel.

By Riju — about