Cache Stampede and Thundering Herd: Request Coalescing, TTL Jitter and Stale-While-Revalidate (2026)

A cache is a promise that the database will only see a small fraction of your traffic. That promise has a hidden clause: it holds only while the entries are present. The moment one very popular key expires, every request that arrives during the next recompute window discovers the miss at the same instant, and every one of them walks to the database with the same query. A dependency sized for a 2 percent miss rate suddenly receives 100 percent of the traffic for one key, and when that key is expensive to compute, the recompute gets slower under the load, which keeps the key missing, which sends in more requests.

That feedback loop is a cache stampede, and it is the cache-layer cousin of the thundering herd that operating systems and load balancers have fought for decades. It is not exotic: it appears the first time a team adds a TTL to a homepage query, and it reappears every time a deploy flushes Redis. The defences are well understood but rarely combined: request coalescing so that one caller recomputes while the rest wait, TTL jitter so entries do not expire together, probabilistic early expiration so a hot key is refreshed before it dies, and stale-while-revalidate so that expiry never blocks a reader.

What this covers: the mechanics of a stampede with worked arithmetic, how cache-aside, read-through and write-through differ in their exposure, single-flight coalescing in Go and Python, the real hazards of Redis locks and leases, TTL jitter math, the XFetch algorithm, stale-while-revalidate and stale-if-error under RFC 5861, negative caching, warming, CDN collapsed forwarding, the hot-key problem, failure modes, and the metrics that tell you it is happening.

Context and Background

Caching is the cheapest capacity you will ever buy. A well-hit cache turns a 5 ms database read into a 0.3 ms memory read and, more importantly, absorbs the load: if the hit ratio is 98 percent, the database sees one request in fifty. The trouble is that hit ratio is an average, and averages hide correlation. Cache misses do not arrive independently. They arrive in clusters, because the things that cause misses (a TTL boundary, a deploy, a node restart, an invalidation) are shared across many requests.

The terminology is slippery, so it helps to fix it. A cache stampede (also called dog-piling) is specifically the cache-miss case: many concurrent requests for the same missing or expired key all recompute the value. The broader thundering herd is any situation where many waiters are woken or released at once and contend for a resource that can serve only one or a few of them; Linux’s accept() wake-ups and a flood of clients reconnecting after an outage are other members of the family. When people say “thundering herd” in a caching conversation they usually mean the stampede, and this article uses the two terms interchangeably where the mechanism is the same, noting where it is not.

Three realities make the problem harder than it looks. First, the cost of a miss is not constant. A cold lookup that joins five tables, calls two downstream services and renders a template may take two seconds while a warm one takes a millisecond, so the miss window is long and the stampede cohort is large. Second, the load on the origin is superlinear in the miss window: the longer the recompute takes, the more requests pile in, and the more they slow the recompute. Third, caches are usually shared by many keys with very different popularity, so a single hot key can drive an outsized share of the damage while the aggregate hit ratio still looks healthy.

The mitigation space has been explored in both industry and academia. Facebook’s memcache deployment, described in the NSDI 2013 paper “Scaling Memcache at Facebook”, introduced leases to stop both stale sets and thundering herds: on a miss the server hands one client a token that permits it to fill the key, and other clients are told to wait briefly or use slightly stale data. Vattani, Chierichetti and Lowenstein published “Optimal Probabilistic Cache Stampede Prevention” at VLDB 2015, which gave a lock-free randomised refresh rule with a provable optimality argument. On the HTTP side, RFC 5861 defined stale-while-revalidate and stale-if-error so that shared caches can serve stale content while refreshing in the background, and current HTTP caching semantics live in RFC 9111.

If you have read our posts on retries, backoff with jitter and retry budgets you will recognise the shape: a correlated burst that arrives exactly when the dependency is weakest. The same toolkit (spread in time, cap concurrency, serve degraded answers) applies here, and a stampede on the database is often the first domino in the retry storm that article describes.

Anatomy of a Cache Stampede: Reference Architecture

Direct answer: a cache stampede happens when a popular entry expires or is evicted and many concurrent requests all miss and recompute the same value at once. The reference defence is layered: coalesce duplicate misses per process, elect one refresher per key across processes with a lease, stagger expiry with jitter, refresh early for hot keys, and serve stale data while the refresh runs.

Figure 1: A layered cache stampede defence. A miss is first collapsed inside the process by single-flight, then arbitrated across processes by a per-key lease; losers serve stale data or wait, and the single winner writes the value back with a jittered TTL.

The diagram shows the order of the defences. The cheapest check is the cache itself. A miss does not go straight to the database; it passes through an in-process single-flight group, so the thousand goroutines or threads inside one application server collapse into one call. The surviving call then competes for a cross-process lease so that, of fifty application servers, one does the work. The remaining servers do not hammer the database; they either wait a short bounded time for the winner’s write or return a stale copy if policy allows. The winner finally writes the fresh value with a TTL that carries random jitter so its expiry does not align with its neighbours.

The mechanics, with numbers

Take a single key serving 1,000 requests per second, whose value takes 2 seconds to compute at the origin. In the steady state the key is cached and the origin sees nothing. At the instant of expiry, with no protection, every request arriving for the next 2 seconds finds a miss and starts its own recompute: that is 1,000 × 2 = 2,000 concurrent identical queries. If the database can run 200 such queries concurrently at full speed, the other 1,800 queue, and each query’s latency stretches. Suppose contention doubles the recompute time to 4 seconds. Now the window is 4 seconds and 4,000 requests have joined. The relationship is a feedback loop: arrivals during the window equal the rate times the window, and the window grows with the arrivals.

Contrast this with perfect coalescing. One request recomputes, 1,999 wait, and the origin sees exactly one query. The cost is that the waiters experience up to 2 seconds of added latency, which is why the next step is to avoid making them wait at all. These numbers are illustrative, not measurements, but the structure (arrivals equal rate times window, with the window itself load-sensitive) is what makes stampedes self-reinforcing.

Figure 2: The failure sequence. After a hot key expires, each client independently observes a miss and independently recomputes. The origin performs N identical computations and the cache receives N identical writes, of which N minus one are wasted.

The three cache patterns and their exposure

How your application talks to the cache determines where the herd forms.

In cache-aside (lazy loading), the application reads the cache, and on a miss it loads from the database and writes the entry back. It is the most common pattern and the most exposed, because every application instance independently decides to recompute and nothing in the pattern coordinates them. Stampede defence has to be bolted on by the caller.

In read-through, the cache layer itself owns the loader. The application asks the cache for a key and the cache fetches from the source on a miss. Because the loader lives in one place, coalescing is natural to implement: Caffeine in Java and many managed caches serialise concurrent loads for the same key inside the cache. The exposure then moves to the cache implementation, and you should confirm that it does coalesce rather than assume it.

In write-through, writes go to the cache and the store together, so the cache is warm for anything that was written. This shrinks the miss window for freshly written data but does nothing for expiry or eviction of read-mostly entries, and it adds write latency. In write-behind, the cache absorbs writes and flushes later; it has the same read-side exposure and adds durability risk. None of the write patterns removes the need for stampede defences on the read path, which is where the herd forms.

Why the miss is correlated

Understanding why misses cluster tells you which defence to reach for. There are four common sources of correlation. The first is synchronized expiry: a batch job, a deploy or a cache warmer populates many keys at once with the same TTL, and they all expire at once. The second is hot-key expiry: one key carries so much traffic that its individual expiry is itself a stampede. The third is cold start: a cache flush, a failover to an empty replica or a new region brings the hit ratio to zero for every key at the same time. The fourth is eviction pressure: when memory is full, an eviction policy removes entries, and a burst of evictions of popular keys looks like a burst of expiries.

Each source has a matching remedy. Synchronized expiry calls for jitter. Hot-key expiry calls for early refresh or stale serving. Cold start calls for warming and admission control. Eviction pressure calls for capacity planning and a policy that protects hot entries. The rest of this article takes each in turn.

Request Coalescing, Locks and Leases in Practice

Request coalescing means that when several callers ask for the same key at once, exactly one of them does the work and the others share its result. It is the single highest-value defence because it is cheap, local and needs no coordination infrastructure. It also has a precise scope: it collapses duplicates within one process (or one proxy), not across a fleet. A fleet of 50 servers with perfect in-process coalescing still sends 50 identical queries to the database. That is a 40-fold improvement over 2,000, and often enough, but it is worth knowing where the floor sits.

Single-flight in Go

Go’s golang.org/x/sync/singleflight package is the canonical implementation. A Group tracks in-flight calls by string key. Do(key, fn) runs fn for the first caller; any caller that arrives with the same key while the call is in flight blocks and receives the same result and error, with a boolean telling it whether the value was shared. DoChan returns a channel instead of blocking, which lets you combine it with a select on a context deadline. Forget(key) drops the key so the next caller starts a fresh call rather than joining a slow one.

package cache

import (
    "context"
    "time"

    "golang.org/x/sync/singleflight"
)

type Loader func(ctx context.Context, key string) ([]byte, error)

type Cache struct {
    store  Store // Redis, memcached, in-memory...
    group  singleflight.Group
    load   Loader
    ttl    time.Duration
}

func (c *Cache) Get(ctx context.Context, key string) ([]byte, error) {
    if v, ok := c.store.Get(ctx, key); ok {
        return v, nil
    }
    // The shared call must not be tied to one caller's context,
    // or a single cancelled request fails the whole herd.
    ch := c.group.DoChan(key, func() (interface{}, error) {
        lctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
        defer cancel()
        v, err := c.load(lctx, key)
        if err == nil {
            c.store.Set(lctx, key, v, jitter(c.ttl, 0.1))
        }
        return v, err
    })
    select {
    case res := <-ch:
        if res.Err != nil {
            return nil, res.Err
        }
        return res.Val.([]byte), nil
    case <-ctx.Done():
        return nil, ctx.Err()
    }
}

Two details in that snippet matter more than the call itself. The shared function runs under its own bounded context rather than the first caller’s context: if it inherited the first caller’s context and that client disconnected, the cancellation would propagate to every waiter, turning one impatient client into a herd-wide failure. And each waiter selects on its own context, so a caller with a tight deadline gives up on its own without cancelling the shared load. A third subtlety is error sharing: if the load fails, all waiters receive the same error. That is usually right, but a transient failure now fails a whole cohort at once, so pair single-flight with a short negative-result cache or a retry that goes through the same group.

The same idea in Python

In asyncio the equivalent is a dictionary of in-flight futures. The first caller creates the task and registers it; later callers await the same task.

import asyncio, random, time

class Coalescer:
    def __init__(self, loader, store, ttl=300, jitter=0.1):
        self.loader, self.store = loader, store
        self.ttl, self.jitter = ttl, jitter
        self._inflight: dict[str, asyncio.Task] = {}

    async def get(self, key):
        hit = await self.store.get(key)
        if hit is not None:
            return hit
        task = self._inflight.get(key)
        if task is None:
            task = asyncio.create_task(self._fill(key))
            self._inflight[key] = task
            task.add_done_callback(lambda _t, k=key: self._inflight.pop(k, None))
        # shield so one cancelled waiter does not cancel the shared task
        return await asyncio.shield(task)

    async def _fill(self, key):
        value = await self.loader(key)
        ttl = self.ttl * (1 + random.uniform(-self.jitter, self.jitter))
        await self.store.set(key, value, ex=int(ttl))
        return value

The asyncio.shield call plays the same role as the separate context in Go. Without it, cancelling one awaiting request would cancel the underlying task and strand every other waiter.

Cross-process coalescing: distributed locks and leases

To reach the floor of one recompute per key across the fleet you need a cross-process arbiter. The simplest is a short-lived lock in Redis, created with SET lock:key token NX PX 5000: it succeeds only if the key does not exist and expires automatically after 5 seconds so a crashed holder cannot block forever. The winner recomputes and then releases the lock, ideally with a small script that deletes it only if the stored token still matches, so it cannot release a lock that has since been acquired by someone else. Losers poll the cache for a short interval, or return stale data.

This is the same idea as the memcache lease from the NSDI 2013 Facebook paper: a lease is a lock that carries a token and a time limit, and the token also lets the server reject a stale write. In the paper’s description the server returns a token on a miss and, by default, hands out a token only once in a bounded interval per key, so most clients that miss in the meantime are told to retry a little later or use slightly stale data. A lease therefore does two jobs at once: it elects one filler, and it prevents a delayed filler from overwriting a newer value.

Locks around caches come with hazards that are easy to underestimate.

Lock expiry before completion. If the recompute takes longer than the lock TTL, the lock expires while the holder is still working. A second client acquires it and starts a second recompute, and the system has degraded to a partial stampede. Setting the TTL very long fixes this but makes a crashed holder block refresh for that long. A reasonable approach is a lock TTL of a few multiples of the p99 recompute time plus a bounded wait on the loser side, and treating lock-held-too-long as a metric.

Lost wake-ups and polling. Losers that poll in a tight loop convert the lock into a new source of load, a cache read storm instead of a database one. Poll with jitter and a small cap (for example 10 to 50 ms steps, bounded to a total wait that matches your latency budget), or use pub/sub to notify waiters, accepting the added complexity.

Fencing and correctness. A lock held by a process that is paused (a long garbage collection, a VM stall) can expire and be taken by another while the first still believes it holds it. Martin Kleppmann’s analysis of Redis Redlock makes the general point: locks with time-based expiry cannot guarantee mutual exclusion without a fencing token that the protected resource checks. For a cache, the stakes are lower than for a financial ledger, because a duplicate recompute writes the same value, so a best-effort lock is acceptable. Do not extend that reasoning to a case where the recompute has side effects.

Single point of contention. All lock traffic for a hot key lands on one Redis shard. For truly hot keys, the lock itself becomes the hot key, which is one reason to prefer the stale-serving and probabilistic techniques below, which need no lock at all.

A pragmatic combination is a two-level scheme: in-process single-flight to collapse local duplicates, then a best-effort Redis SET NX PX lock to elect one process, with losers serving stale data rather than polling.

TTL Jitter: Breaking Synchronized Expiry

Jitter is the cheapest defence and the one most often forgotten. If a warmer or a deploy loads 100,000 keys with an identical 300-second TTL at time zero, all 100,000 expire in the same instant, five minutes later. The origin sees the full 100,000 recomputes in one burst, then again at 600 seconds if each refresh restores the same TTL, and the pattern persists as a perpetual comb of load.

Add randomness to the TTL. If each key’s TTL is drawn uniformly from 300 ± 10 percent, that is from 270 to 330 seconds, the expiries spread over a 60-second window. The peak rate falls from 100,000 in one instant to roughly 100,000 / 60 ≈ 1,667 per second, about a 60-fold reduction in peak, with the same average hit ratio. Widen the band to ± 20 percent and the window becomes 120 seconds, a peak of about 833 per second. The mean TTL is unchanged because the distribution is symmetric, so you give up no freshness on average.

Choose the jitter width by considering two constraints: it must be wide enough that the resulting recompute rate stays under the origin’s headroom, and narrow enough that the worst-case staleness is still acceptable to the business. A useful way to size it: take the number of co-expiring keys N, the origin’s sustainable recompute throughput R per second, and require the window W ≥ N / R. If your origin can sustain 500 recomputes per second and you have 100,000 keys that load together, you need W of at least 200 seconds, which for a 300-second TTL means roughly ± 33 percent. If that is too wide, the better fix is to stagger the loading itself (warm in batches) rather than to widen jitter.

import random

def jittered_ttl(base_seconds: int, spread: float = 0.1) -> int:
    """Uniform jitter around base. spread=0.1 gives +-10 percent."""
    return max(1, int(base_seconds * (1 + random.uniform(-spread, spread))))

Jitter has two limits. It does nothing for a single hot key, whose lone expiry is itself the stampede. And it does not help cold start, where the cache is empty rather than expiring. For those you need the next techniques.

Probabilistic Early Expiration and Stale-While-Revalidate

Coalescing makes the herd small; jitter makes it infrequent. Neither removes the moment at which one reader must wait for a recompute. The next two techniques attack that wait directly, one by refreshing before expiry and one by letting readers use expired data.

Probabilistic early expiration (XFetch)

The VLDB 2015 paper by Vattani, Chierichetti and Lowenstein proposes a rule that needs no lock and no coordination. Each cached entry stores its expiry time and a value delta, the measured time it took to compute the value. On every read, the reader flips a biased coin: it recomputes early if

now - delta * beta * ln(rand()) >= expiry

where rand() is uniform in (0, 1], so ln(rand()) is negative and the term adds a random positive lead time ahead of expiry. The parameter beta defaults to 1 and can be raised to refresh earlier. The effect is that the closer the entry gets to expiry, the more likely any given reader is to volunteer, and the longer the recompute takes (larger delta), the earlier volunteering begins. The paper’s argument is that this randomised rule is optimal in a precise sense for preventing stampedes without coordination; I am relying on the published abstract for that claim and am not reproducing the proof here.

The probability arithmetic is easy to work out yourself. The chance that a single read at t seconds before expiry triggers a refresh equals the chance that -delta * beta * ln(U) >= t, which is exp(-t / (delta * beta)). With delta = 2 seconds and beta = 1:

Seconds before expiry	Per-read refresh probability
10	exp(-5) ≈ 0.67%
6	exp(-3) ≈ 5.0%
2	exp(-1) ≈ 36.8%
0	100%

Now consider traffic. With a request rate of λ per second, the expected number of refresh triggers per second at t seconds before expiry is λ times that probability. Integrating the hazard gives the time at which the first volunteer is expected: the cumulative expected triggers reach 1 when exp(-t / delta) ≈ 1 / (λ · delta), that is t ≈ delta · ln(λ · delta). For our hot key at 1,000 requests per second and delta = 2, that is 2 × ln(2000) ≈ 15 seconds before expiry. For a key receiving 1 request per second, it is 2 × ln(2) ≈ 1.4 seconds. This is the elegant property: hot keys refresh early automatically, and cold keys, which cannot stampede, are left to expire close to their deadline and cost nothing extra. The numbers are derived from the formula above, not from a benchmark.

import math, random, time

def xfetch_get(cache, key, ttl, recompute, beta=1.0):
    entry = cache.get(key)            # {"value", "delta", "expiry"} or None
    now = time.time()
    if entry is not None:
        # random.random() is in [0,1); use 1 - r to stay in (0,1]
        gap = entry["delta"] * beta * -math.log(1.0 - random.random())
        if now + gap < entry["expiry"]:
            return entry["value"]          # still comfortably fresh
    start = time.time()
    value = recompute(key)
    delta = time.time() - start
    cache.set(key, {"value": value, "delta": delta, "expiry": time.time() + ttl},
              ex=int(ttl + 60))            # physical TTL longer than logical expiry
    return value

Note the last line: the entry is physically stored longer than its logical expiry, so that an early refresher still finds the old value to serve while it works. Also note that several readers can still volunteer in the same instant; XFetch lowers that probability sharply but does not make it zero, which is why it composes well with single-flight on the recompute path.

Stale-while-revalidate and stale-if-error

Where XFetch refreshes before expiry, stale-while-revalidate (SWR) keeps serving after it. RFC 5861 defines two Cache-Control extensions. stale-while-revalidate=N says that a cache may serve a stale response for up to N seconds past its freshness lifetime while it revalidates in the background. stale-if-error=N says that a cache may serve a stale response for up to N seconds if the origin returns an error, such as a 5xx or a connection failure. A typical header looks like this:

Cache-Control: max-age=60, stale-while-revalidate=300, stale-if-error=86400

That reads as: fresh for a minute; for the next five minutes serve it instantly and refresh in the background; and if the origin is failing, keep serving it for up to a day. The RFC was published in 2010 as an independent submission and is widely implemented by CDNs and reverse proxies. HTTP caching in general is now specified in RFC 9111. Browser support for these directives has been uneven historically, so check your target clients rather than assuming; the shared-cache behaviour is where they matter most for stampedes.

Figure 3: The combined read path. Fresh entries are served, a probabilistic coin may trigger an early recompute, expired entries inside the SWR window are served stale while exactly one background refresh runs, and only entries beyond the window force a blocking, coalesced miss.

The same pattern works in application caches, where it is sometimes called “soft TTL and hard TTL”. The entry carries a soft expiry (after which it is stale but usable) and a hard expiry (after which it is gone). A read between the two returns the value immediately and, guarded by a per-key flag or single-flight, triggers one asynchronous refresh. The reader never waits except on a true cold miss. The cost is that users can see data up to soft-plus-window seconds old, and you must decide, per data type, whether that is acceptable. A product description can be five minutes stale; an account balance or an inventory count at checkout generally cannot.

Two edge cases deserve a mention. A background refresh that fails must not silently extend staleness forever: bound the total stale window and alert when refreshes fail. And with stale-if-error you are trading correctness for availability on purpose, so make sure the trade is documented for the data in question.

Putting the techniques side by side

Technique	Removes which cost	Needs coordination	Typical added staleness	Weak spot
In-process single-flight	Duplicate recomputes inside one process	No	None	One query per process still reaches origin
Redis lock or lease	Duplicate recomputes across fleet	Yes, a shared store	None, but losers wait	Lock expiry, hot-key contention
TTL jitter	Synchronized mass expiry	No	Up to the jitter band	Does not help a single hot key
XFetch	Reader-visible miss on hot keys	No	None (refreshes early)	Needs stored delta, rare duplicate volunteers
Stale-while-revalidate	Reader waiting on a refresh	No	Up to the stale window	Needs tolerance for stale data
Request collapsing at CDN	Duplicate origin fetches at the edge	Built into the CDN	None	Behaviour is vendor specific

No single row is sufficient. The production pattern is jitter and single-flight everywhere, XFetch or SWR for keys you know are hot, and a lease where the recompute is expensive enough that even one-per-process duplication hurts.

Negative Caching, Warming and the Edge

Negative caching and the penetration variant

A related failure is cache penetration: requests for keys that do not exist are never cached, because there is nothing to store, so each one reaches the database. An attacker or a buggy client requesting random identifiers turns the cache into a pass-through. The remedy is negative caching, storing the fact of absence with a short TTL (for example 30 to 60 seconds) so repeated misses for the same nonexistent key are absorbed. Keep the negative TTL short and invalidate on creation, or a newly created record will appear missing. For large, unpredictable key spaces, a Bloom filter in front of the database can answer “definitely not present” cheaply; our guide to probabilistic data structures such as Bloom and cuckoo filters covers the false-positive trade-off, which for this use is benign because a false positive just costs one database lookup.

Negative entries should also be coalesced and jittered like positive ones, and you should count them separately in metrics so a surge of absent-key traffic is visible.

Cache warming and surviving a cold start

After a flush, a failover to an empty replica, or a new region launch, the hit ratio is zero across the board, and no per-key technique helps because every key is a miss at once. The defences are operational. Pre-warm from a snapshot (Redis RDB or AOF persistence retains data across restarts, and replicas can be promoted warm), replay recent access logs against the cache before shifting traffic, or warm the top-N keys by popularity first. Where warming is not possible, shed or rate-limit at the origin so it survives the transition, and ramp traffic gradually, a technique that fits naturally with the admission control described in our post on distributed rate limiting. A load balancer that ramps weights slowly onto a cold node serves the same purpose; see the notes on slow start in load balancing algorithms.

Warm in batches with jittered TTLs. A warmer that loads everything at once with an identical TTL has just manufactured the synchronized-expiry problem it was meant to prevent.

CDN collapsed forwarding and origin shielding

At the HTTP edge the same problem appears as many users requesting one URL as it expires on a point of presence. Most mature CDNs and reverse proxies implement request collapsing, also called collapsed forwarding: while one request to the origin for a URL is in flight, other requests for that URL wait and share the response. Varnish does this with its waiting list for objects being fetched, and NGINX offers proxy_cache_lock to allow only one request to populate a cache element, together with proxy_cache_use_stale updating and proxy_cache_background_update to serve stale content while a background request refreshes it. Check your vendor’s documentation for the exact defaults and timeouts, since they differ and some features are off by default.

An origin shield (a designated intermediate cache tier) extends this: edge nodes miss to the shield, not directly to the origin, so the origin sees one request per object per shield rather than one per edge location. Coalescing at two levels, edge and shield, can reduce a worldwide stampede to a handful of origin fetches.

Figure 4: Where each defence lives. Collapsed forwarding at the CDN edge and shield, single-flight in each application server, and a lease on the shared Redis tier together reduce a worldwide burst of misses to a trickle of database queries.

The hot-key problem

Some keys are simply too hot for one cache shard. A celebrity profile, a flash-sale product or a viral page can send hundreds of thousands of requests per second to one key, which lives on one Redis shard or one memcached server. Even with perfect stampede prevention the shard’s network or CPU saturates. Three tactics help. Replicate the hot key under several names (key#1 to key#8) and let clients pick one at random, spreading load across shards at the price of invalidating all copies on update. Add a small local in-process cache with a short TTL (one to five seconds) in front of the remote cache, which absorbs most reads for a hot key at the application server. And use server-assisted client-side caching: since Redis 6, the CLIENT TRACKING feature lets the server remember which keys a client has cached and send invalidation messages when they change, so clients can hold hot values locally and stay consistent. Facebook’s memcache deployment likewise replicated popular keys across clusters.

A local cache fed by a remote one gives you a second, cheap layer of single-flight and removes the stampede for that key from the shared tier altogether. The cost is bounded staleness equal to the local TTL, plus memory on every application server.

Trade-offs, Gotchas, and What Goes Wrong

Every defence above shifts a cost rather than deleting it, and several have failure modes of their own.

Coalescing turns a slow origin into a slow herd. If one recompute hangs, every waiter hangs with it. Always bound the shared call with its own timeout, bound each waiter with its own deadline, and decide what waiters do on timeout: fail fast, serve stale, or fall back to a degraded default. Use Forget in Go, or an equivalent, so a stuck flight does not block all subsequent callers for that key indefinitely.

Lock expiry and the half-stampede. A lock that expires before the holder finishes re-admits the herd in a smaller form. Track lock acquisition failures and lock-held duration against the recompute p99. If the p99 recompute time creeps toward the lock TTL, you are one slow day from a stampede.

Stale serving hides outages. Stale-if-error is wonderful until the origin has been broken for six hours and nobody noticed because every user still gets an answer. Export the age of served content and the count of stale serves, and alert on both. The mitigation of one failure must not become the concealment of another.

Eviction storms. When memory fills, the eviction policy removes entries on its own schedule. Under an LRU-like policy, a scan or a large batch read can flush the hot working set and cause a wave of misses that looks like a stampede with no expiry involved. Treat memory headroom as a stampede control, size the cache for the working set with margin, and choose a policy that resists scans.

Cold start after a flush. A FLUSHALL during an incident, a failover to an empty replica, or a deploy that changes the cache key format all produce a global cold cache. Version your key format deliberately, avoid flushing in production, and have a warm-up and traffic-ramp procedure ready before you need it.

Early refresh costs real work. XFetch and background refresh recompute values that might never be read again. For a catalogue of millions of keys, refreshing proactively all of them wastes capacity. Apply probabilistic or scheduled refresh selectively, to keys whose access rate and recompute cost justify it, and let the long tail expire lazily.

Retry amplification. A stampede that makes the database slow causes timeouts, and timeouts cause client retries, and retries are additional misses on a hot key. This is the connection to the metastable failures discussed in the retries article: stampede prevention should be paired with retry budgets and deadlines so that the cache’s failure does not become the retry layer’s amplifier. Our guide to retries, backoff with jitter and circuit breakers covers that half.

Consistency. Every technique here widens the window in which a reader can see an old value. If your data has correctness requirements (prices at checkout, permissions, inventory), classify each cached object and pick the weakest defence that still honours its freshness contract. Permissions in particular should rarely be served stale on error.

Anti-patterns

Setting one global TTL for everything is the root cause of most synchronized-expiry incidents. Using a very long lock TTL to avoid lock expiry just moves the failure to crashed holders. Calling the loader inside a cache lock that spans the whole cache (not per key) serialises unrelated misses and destroys throughput. And relying on a cache-hit-ratio dashboard alone: ratio is an average that stays at 98 percent while one key melts the database.

Monitoring: signals that a stampede is happening

You want to see the herd forming before the database pages you. Useful signals include:

• Per-key miss rate and concurrent loaders per key, sampled for the top keys. More than one loader per key per process means coalescing is not working.
• Origin query rate versus cache request rate, plotted together. In a stampede the origin curve spikes while the request curve does not.
• Recompute latency (p50, p99) per loader, because rising recompute time is the leading indicator of the feedback loop.
• Lock contention: acquisition failures, wait time, and lock-held duration against TTL.
• Stale serves and age of served content, so degraded mode is visible.
• Eviction rate and memory pressure, to catch eviction storms.
• Expiry histogram: the number of keys expiring per second across the next minutes. A spike here predicts a synchronized-expiry event, and it is cheap to compute from a sampled scan.

Build one dashboard that places these next to database CPU and connection counts. In incident reviews the sequence is usually clear in hindsight: a miss spike, then origin latency, then timeouts, then retries.

Practical Recommendations

Start with the changes that are cheap and universal, then add the ones that match your hot keys.

First, put jitter on every TTL you set, without exception, and make it a property of the cache client library rather than a convention each team must remember. A band of ± 10 to 20 percent is a sensible default; size it against the arithmetic of co-expiring keys and origin throughput shown earlier.

Second, add in-process request coalescing around every cache-miss loader. In Go use singleflight with a detached, bounded context for the shared call; in Python or Node, share the in-flight promise or task and shield it from individual cancellation.

Third, introduce soft and hard TTLs, or HTTP stale-while-revalidate, for data where seconds or minutes of staleness are tolerable. This removes the reader wait for most expiries.

Fourth, for the handful of keys you know to be hot, use XFetch-style early refresh or a leased background refresher, and replicate or locally cache them to avoid shard saturation.

Fifth, protect the origin directly: a concurrency limit or rate limit on the loader path, a circuit breaker, and a retry budget, so that when every defence leaks the origin degrades rather than collapses.

Sixth, rehearse the cold start. Flush a staging cache under load and watch what happens; then write the warming and ramp procedure.

Checklist:

• [ ] Every cache write sets a TTL with jitter, applied centrally.
• [ ] Every miss loader is wrapped in single-flight with its own timeout and detached context.
• [ ] Hot keys are identified from metrics and have a documented strategy (early refresh, stale serving, local cache or replication).
• [ ] Each cached data class has a stated maximum staleness and a stale-on-error policy.
• [ ] Negative results are cached briefly and counted separately.
• [ ] Lock or lease TTLs exceed the p99 recompute time, with bounded loser waiting.
• [ ] The origin has a concurrency limit and a load-shedding path.
• [ ] Dashboards show per-key miss rate, loader concurrency, stale serves and expiry histogram.
• [ ] A cache cold-start drill has been run in the last quarter.
• [ ] CDN collapsed forwarding and origin shielding are configured and verified, not assumed.

Frequently Asked Questions

What is a cache stampede?

A cache stampede, also called dog-piling, occurs when a popular cache entry expires or is evicted and many concurrent requests all miss and recompute the same value simultaneously. The result is a sudden spike of identical queries against the origin, which can slow or overload it. Because slower recomputes keep the key missing for longer, the effect can become self-reinforcing until the load is removed or the entry is finally written back.

What is the difference between a cache stampede and a thundering herd?

A thundering herd is the general phenomenon where many waiting processes or clients are released at once and contend for a resource that can serve few of them, such as a socket accept wake-up or mass reconnects after an outage. A cache stampede is the cache-specific case in which the herd is made of requests recomputing one expired key. The mitigations overlap, namely coalescing, jitter and admission control.

How does request coalescing prevent a stampede?

Request coalescing lets the first request for a missing key perform the expensive load while later requests for the same key wait and share its result. Within a process this is done by libraries such as Go’s singleflight, and across processes by a lease or lock. The origin sees one query per process, or one per key with a lease, rather than one per request. Always bound the shared call with a timeout.

How do I choose between XFetch and stale-while-revalidate?

Use stale-while-revalidate when slightly old data is acceptable: readers are served immediately and one background refresh runs. Use XFetch when you want the value refreshed before it ever becomes stale, for example for hot keys where staleness is not tolerable, and when you want a coordination-free rule. They compose well: XFetch handles the common case early, and a stale window plus single-flight covers the rare refresh that was missed.

Does TTL jitter fix a cache stampede on a single hot key?

No. Jitter spreads the expiry times of many keys so they do not expire together, which solves synchronized mass expiry. A single hot key still has one expiry moment, and the requests arriving then still collide. For that case combine request coalescing with early refresh such as XFetch, stale-while-revalidate, or a local cache in front of the shared one.

Should I use a distributed lock to stop stampedes?

A best-effort lock, such as Redis SET key token NX PX, is a reasonable way to elect one refresher because a duplicate recompute writes the same value. Be aware of lock expiry before completion, hot-key contention on one shard, and the absence of fencing guarantees. Prefer losers serving stale data over polling, and treat a lock as an optimisation layered on single-flight and jitter, not as a correctness mechanism.

Further Reading

Related posts on this site:

• Retries, exponential backoff with jitter, circuit breakers and retry budgets
• Distributed rate limiting: token bucket, GCRA, Redis and Envoy
• Load balancing algorithms: power of two choices, Maglev, L4 vs L7
• Probabilistic data structures: Bloom filter, HyperLogLog, Count-Min Sketch, cuckoo filter

Primary sources:

• RFC 5861: HTTP Cache-Control Extensions for Stale Content
• Optimal Probabilistic Cache Stampede Prevention (Vattani, Chierichetti, Lowenstein, VLDB 2015)
• Go singleflight package documentation

By Riju — about