LSM Tree Internals: RocksDB Compaction, Write Amplification and Tuning (2026)

Most engineers meet the LSM tree the hard way: a service that ran smoothly for months suddenly shows a p99 write latency of several seconds, the dashboards blame “compaction”, and nobody on the team can say which of the forty RocksDB options matters. The data structure behind that incident is elegant and, at heart, a single bargain: turn random writes into sequential ones now, and pay the sorting bill later in the background. The bill is called compaction, and its size is called write amplification.

This matters in 2026 because log-structured merge trees quietly underpin much of the modern data stack: RocksDB inside TiKV and Kafka Streams, Pebble inside CockroachDB, and Cassandra’s own engine. Understanding the mechanism lets you predict stalls rather than react to them, and size disks and CPUs from first principles. You will leave with a mental model, worked amplification arithmetic, and a tuning checklist.

What this covers: the write and read paths, SSTables and bloom filters, leveled vs tiered (and universal and FIFO) compaction, the RUM trade-off, write stalls and compaction debt, the RocksDB options that matter, a B-tree contrast, failure modes such as tombstone storms, and a practical tuning checklist.

Context and Background

A B-tree updates pages in place. That gives excellent point-read performance and predictable space use, but every small write may dirty a random page, which means random I/O and, on SSDs, internal write amplification from the flash translation layer. The log-structured merge tree, introduced by Patrick O’Neil and colleagues in their 1996 paper The Log-Structured Merge-Tree (LSM-Tree), takes the opposite stance: never update in place. Buffer writes in memory, flush them as immutable sorted files, and periodically merge those files.

Google’s Bigtable paper (2006) popularised the memtable plus SSTable design in production, and LevelDB, written by Jeff Dean and Sanjay Ghemawat, made it a small embeddable library. Facebook forked LevelDB into RocksDB to exploit flash storage and multi-core servers, adding multi-threaded compaction, column families, universal compaction, rate limiting and a long list of tuning knobs. The RocksDB project describes itself as a persistent key-value store for fast storage environments, and its official wiki is the authoritative reference for option semantics.

The engine’s reach is wide. Kafka Streams uses RocksDB as the default local state store. TiKV, the storage layer of TiDB, builds on RocksDB. CockroachDB moved from RocksDB to Pebble, a Go engine that is RocksDB-compatible in on-disk format ideas but written from scratch. Apache Cassandra has its own LSM implementation with its own compaction strategies (size-tiered, leveled, time-window, and the unified compaction strategy added in Cassandra 5.0). If you operate any of these, you operate an LSM tree, and the same physics applies.

Time-series databases are heavy users of the same ideas, with their own twists such as time-partitioned files and columnar blocks. Our deep dive on time-series database internals across InfluxDB, TimescaleDB and QuestDB shows where that family diverges from a general-purpose key-value LSM. And if you are weighing streaming infrastructure that leans on these stores, the comparison of Kafka, RabbitMQ and SQS for task queues is a useful companion.

A vocabulary check before the mechanism. A memtable is the in-memory sorted write buffer. A WAL (write-ahead log) is the sequential on-disk log that makes the memtable durable. An SSTable (sorted string table) is an immutable sorted file of key-value pairs. A sorted run is a sequence of non-overlapping SSTables covering a key range. Write amplification is the ratio of bytes the storage engine writes to the device divided by bytes the application wrote. Read amplification is the number of files or blocks consulted per lookup, and space amplification is on-disk size divided by live data size.

The LSM Tree Reference Architecture

An LSM tree absorbs writes in a memtable, makes them durable through a sequential WAL, flushes full memtables to immutable sorted SSTables in level 0, and then merges those files downward through levels in the background. Reads check the memtable, then SSTables from newest to oldest, using per-file bloom filters to skip files that cannot contain the key.

Figure 1: Write path (left to right and down) and read path in an LSM tree. Writes touch only the WAL and memtable; flush and compaction are background work.

Figure 1 shows the whole machine in one picture. The write path has exactly two synchronous steps: append the record to the WAL and insert it into the memtable. Everything below the memtable is asynchronous. That is why LSM engines sustain high write throughput on both spinning disks and SSDs: the foreground cost is one sequential append plus an in-memory skiplist insert, regardless of how large the database is.

The write path: WAL, memtable, flush

In RocksDB, a Put is added to the WAL (unless disabled) and inserted into the active memtable, by default a skiplist. When the memtable reaches write_buffer_size, it becomes immutable and a new active memtable takes over. A background flush job writes the immutable memtable as an L0 SSTable. The WAL segment covering that memtable can then be retired. The durability knob is the WAL sync policy: syncing every write is safest and slowest, while relying on the operating system to flush (or using manual_wal_flush) trades a short loss window for throughput.

Two options interact here. max_write_buffer_number bounds how many memtables (active plus immutable) may exist, and min_write_buffer_number_to_merge controls how many immutable memtables are merged in memory before a flush. A larger merged flush absorbs overwrites of hot keys before they ever reach disk, which directly lowers write amplification for skewed workloads. The cost is memory and a longer recovery replay of the WAL after a crash.

The on-disk structure: SSTables and levels

An SSTable in RocksDB’s block-based format holds sorted data blocks (4 KB by default, per documented defaults; verify against your version), an index block, optional filter blocks, a properties block and a footer. Each block can be compressed independently with LZ4, Snappy, Zstd or others. Because files are immutable, there is no locking on the read side and no torn pages: a reader holds a reference to a consistent set of files, called a version, while compaction produces new files and atomically swaps the version.

L0 is special. Each flush produces one file whose key range may overlap every other L0 file, because each file simply contains whatever keys arrived in that time window. Levels L1 and deeper are sorted runs: files are non-overlapping and ordered by key. A lookup therefore needs to check every L0 file but only one file per deeper level, found by binary search over file boundaries.

Why reads need bloom filters

Without help, a point lookup for a missing key would probe the memtables, every L0 file, and one file in every deeper level: a handful to dozens of potential block reads. A bloom filter is a compact probabilistic set that answers “definitely not here” or “maybe here”. RocksDB stores a filter per SSTable (or partitioned filter blocks for very large files). A negative answer skips the file with no disk read.

The arithmetic is simple. A bloom filter with m bits for n keys and k hash functions has a false-positive rate of about (1 − e^(−kn/m))^k. At 10 bits per key and an optimal k of about 7, the rate is roughly 0.8 to 1 percent. So a lookup for an absent key across a tree with, say, six candidate files costs on average about 0.06 spurious block reads instead of six real ones. This is why RocksDB documentation commonly recommends around 10 bits per key as a starting point, and why the engine also offers the more space-efficient Ribbon filter at the price of extra CPU on construction.

Bloom filters help point lookups and, with prefix extractors, prefix scans. They do nothing for range scans over arbitrary ranges, which must merge iterators across all overlapping runs. That asymmetry shapes compaction choices, as the next section shows.

The RUM conjecture: the frame for every choice

The RUM conjecture, proposed by Manos Athanassoulis and colleagues at EDBT 2016, argues that an access method that sets tight upper bounds on two of Read overhead, Update overhead and Memory (space) overhead must accept a lower bound on the third. It is a conjecture, not a theorem, but it is an excellent mental compass. A B-tree prioritises read overhead; a pure log prioritises update overhead; compaction policy slides an LSM tree along the curve between them.

Everything in the rest of this article, from leveled vs tiered compaction to bloom filter sizing to the size of the block cache, is a choice of where to stand on that curve for your workload. There is no free corner.

How Compaction Works and Why Write Amplification Happens

Compaction merges sorted runs, discarding overwritten values and deleted keys, and writes new SSTables. Each time a key is rewritten as it moves down the tree, write amplification increases. The compaction policy decides how often data is rewritten, and so sets the balance between write amplification, read amplification and space amplification.

Figure 2: Leveled compaction keeps one sorted run per level; tiered compaction accumulates several runs per tier before merging them.

Leveled compaction: low read and space amplification

In RocksDB’s default leveled compaction, each level from L1 down is a single sorted run with a size target that grows by max_bytes_for_level_multiplier (commonly 10). When L(n) exceeds its target, the compaction picker chooses a file, finds the overlapping files in L(n+1), merges them, and writes new files into L(n+1). Because each level holds one run, a point lookup consults at most one file per level, and in steady state roughly 90 percent of the data sits in the last level, so space amplification is low, around 1.1x in the ideal case.

The price is write amplification. A key written into L1 will, over its life, be rewritten as it descends to each deeper level. When a file from L(n) is merged into L(n+1) with a size ratio of T, it overlaps about T files at the next level on average, so each byte pushed down drags roughly T bytes of existing data through the merge. A common back-of-envelope estimate is therefore write amplification of about T per level plus one for the flush, times the number of levels the data traverses. The RocksDB wiki itself says only that leveled write amplification is “often larger than 10”, and treats exact figures as workload-dependent.

Worked example: arithmetic of write amplification (illustrative)

Take a database that stabilises at 1 TB of live data, with max_bytes_for_level_base at 256 MB and a multiplier of 10. With the dynamic level sizing that RocksDB has defaulted to since version 8.4 (level_compaction_dynamic_level_bytes), the last level is sized to hold the bulk of the data and upper levels are derived from it by dividing by the multiplier. The levels then look like this (illustrative figures):

Level	Approx. size	Role
L6 (last)	~900 GB	~90% of data
L5	~90 GB
L4	~9 GB
L3	~900 MB
L2	~90 MB
L1	~9 MB or base
L0	a few flushed files

Data reaches the last level after crossing roughly five or six compactions. Using the rule of thumb that each cross-level compaction rewrites about T/2 to T bytes per byte pushed (the lower end reflects the fact that a pushed file does not always find a full set of neighbours, the upper end the worst case), we get between about 5 x 5 = 25 and 5 x 10 = 50 bytes written per user byte before counting the WAL and the flush. Real systems often report single digits to the low tens depending on key distribution, because overwritten keys die before descending and because many compactions are partial. Treat 10 to 30 as a planning range for random-key write-heavy workloads on 1 TB of data, then measure.

The consequence on hardware is concrete. If the application writes 20 MB/s of random keys with a write amplification of 20, the storage device sees 400 MB/s of writes, plus the reads needed for compaction. On a flash drive rated for a given number of drive writes per day, the lifetime shrinks by the same factor. This calculation (user rate x WA = device write rate) belongs in every capacity plan. RocksDB exposes the real numbers: the rocksdb.compact.write.bytes and rocksdb.flush.write.bytes tickers and the compaction stats dump in the LOG file let you compute your actual write amplification rather than guess.

Tiered (size-tiered) compaction: low write amplification

Size-tiered compaction, used by Cassandra’s STCS and by HBase-style engines, accumulates several similarly sized runs and merges them into one larger run once a threshold count is reached. Each byte is rewritten once per tier, and the number of tiers grows logarithmically with the fan-in, so write amplification scales roughly with the number of tiers (log base T of the data-to-memtable ratio) rather than with T times that. That is a large saving: where leveled pays T per level, tiered pays about 1 per tier.

The cost moves to reads and space. A lookup may have to examine one file in every run of every tier, up to T runs per tier, so read amplification grows by roughly a factor of T, which bloom filters partially mask for point lookups but not for scans. Space amplification is worse because obsolete versions live in multiple runs until the next merge, and the largest merge needs temporary space close to the size of its inputs. A widely cited figure is that size-tiered strategies can need on the order of twice the data size in headroom in the worst case; plan for it rather than hoping for it.

Universal compaction in RocksDB: tiering with guardrails

RocksDB implements the tiered idea as universal compaction. Per the RocksDB wiki, it treats all runs as a sequence ordered by age and decides what to merge using a size-ratio rule (compaction_options_universal.size_ratio), minimum and maximum merge widths, and a space-amplification trigger (max_size_amplification_percent) that forces a full compaction when estimated space amplification crosses a threshold. The wiki estimates space amplification as the sum of all runs except the oldest, divided by the oldest, and notes that a major compaction can temporarily double disk usage.

The wiki also gives a deliberately rough write-amplification model for universal compaction and admits there is no reliable mathematical model yet. In its worked example (25 percent space amplification limit, 256 MB flushes, 256 GB oldest run, trigger of 11) the estimate lands near 9. That is the right level of confidence to hold: universal can deliver markedly lower write amplification than leveled on write-heavy workloads, but you must benchmark your own data.

FIFO compaction: when data expires anyway

FIFO compaction does not merge at all. It simply deletes the oldest SSTable files when total size (or, with TTL options, age) exceeds a limit. It suits cache-like and short-retention telemetry data where write amplification should approach one and you never read old data back. It cannot handle updates or deletes in any meaningful way, and range scans are expensive because files overlap. Think of it as a ring buffer that happens to use SSTables.

Partial compaction and file picking

Modern RocksDB leveled compaction does not merge whole levels. The picker selects a file by compaction_pri policy and merges only that file with its overlapping neighbours. The default, kMinOverlappingRatio, prefers the file whose overlap with the next level is smallest relative to its own size, which directly minimises bytes rewritten per byte pushed. Other policies such as kByCompensatedSize and kOldestSmallestSeqFirst exist for specific skew patterns, for example when cold ranges never get compacted and keep stale data alive.

Compaction can also run in parallel across levels and, with subcompactions (max_subcompactions), a single large compaction can be split by key range across threads. max_background_jobs is the umbrella limit on concurrent flush and compaction work; RocksDB divides it between flush and compaction threads. The default is conservative, so on a many-core NVMe machine raising it is often the first, cheap lever when compaction falls behind.

Hybrid and adaptive strategies

Research has shown the leveled/tiered dichotomy is a spectrum. The Dostoevsky design (Dayan and Idreos, SIGMOD 2018) proposes lazy leveling, with tiering at upper levels and leveling only at the last level, which captures most of the write savings of tiering while keeping space amplification and point-lookup cost close to leveling. Monkey (Dayan, Athanassoulis, Idreos, SIGMOD 2017) showed that giving shallower levels more bloom filter bits per key than deeper levels lowers aggregate lookup cost for the same memory, because the bulk of the data sits at the last level where a false positive costs the same as anywhere else. Cassandra’s unified compaction strategy generalises in a similar direction by exposing a scaling parameter that moves continuously between tiered and leveled behaviour.

WiscKey (Lu et al., FAST 2016) attacks write amplification differently, by separating large values from keys. Only keys and value pointers are sorted and compacted; values go to an append-only value log. RocksDB’s BlobDB (integrated blob files) follows the same principle: with large values, compaction rewrites only the small key portion. The cost is garbage collection of the blob files and an extra read per lookup.

Deeper Analysis: Reads, Caches, Stalls and Compaction Debt

The read path, the block cache and the write stall machinery are where an LSM tree’s theory meets production. Reads are cheap when filters and caches absorb them; writes are cheap only while background compaction keeps up. When it does not, RocksDB deliberately slows or stops writers to protect read performance and disk space.

The point lookup, step by step

Figure 3: A point lookup walks memtables, every L0 file, then one file per deeper level. Bloom filters and the block cache decide whether any disk I/O happens.

The lookup first checks the active and immutable memtables. On a miss it checks L0 files from newest to oldest, since any of them might hold the freshest version. For each deeper level it binary-searches file boundaries to find the single candidate file, consults that file’s bloom filter, and only on a “maybe” reads the index block and then the data block.

Two caches matter. The block cache (block_cache, an LRU or the newer HyperClockCache) holds uncompressed data blocks, and by setting cache_index_and_filter_blocks you can also keep index and filter blocks in it, putting all memory under one budget. The OS page cache holds compressed file pages. A well-tuned deployment decides deliberately how much RAM goes to the block cache and how much is left for the page cache, because the same hot block may be cached twice, once in each form.

A worked read-amplification example (illustrative): an L0 trigger of 4 means up to 4 L0 files plus 5 deeper levels, so a worst-case negative lookup could touch 9 files. At a 1 percent false-positive rate the expected number of wasted block reads is about 0.09 per lookup. Without filters it would be nine. For a service handling 50,000 negative lookups per second, that is the difference between about 4,500 spurious reads per second and 450,000. The first fits on one NVMe drive; the second does not fit on anything cheap.

Range scans and iterators have no such shortcut. A forward scan merges one iterator per L0 file plus one per deeper level, so scan cost grows with the number of overlapping runs. This is why tiered compaction hurts scan-heavy workloads more than point-lookup workloads, and why prefix bloom filters and optimize_filters_for_hits (which skips filters on the last level when most lookups are expected to hit) are workload-specific choices rather than universal defaults.

Compaction debt and the three stall triggers

Think of compaction as a debt that writes incur and the background threads repay. If the sustained write rate multiplied by write amplification exceeds the device’s compaction bandwidth, debt grows without bound. RocksDB reports it as estimated pending compaction bytes: how much data compaction would have to rewrite to bring every level back under target.

Figure 4: When writers outrun compaction, debt accumulates, triggers cross thresholds, and writes are throttled or stopped, which appears as tail-latency spikes.

The RocksDB wiki lists three causes of write stalls. First, too many memtables waiting to flush: writes stop when the number of memtables reaches max_write_buffer_number. Second, too many L0 files: level0_slowdown_writes_trigger starts throttling and level0_stop_writes_trigger halts writes. Third, too many pending compaction bytes: soft_pending_compaction_bytes_limit throttles and hard_pending_compaction_bytes_limit stops writes. The wiki does not publish default thresholds on that page, so read your version’s options.h; commonly cited defaults are 20 and 36 L0 files for slowdown and stop, and 64 GB and 256 GB for the soft and hard pending-bytes limits, but verify before relying on them.

The slowdown phase is the dangerous one operationally, because RocksDB applies a delayed write rate (delayed_write_rate) that throttles writers smoothly, so latency degrades gradually and is easy to misread as a network or CPU problem. The tell is in the LOG file and in statistics: “Stalling writes because we have N level-0 files” or “Stopping writes because we have N immutable memtables”. RocksDB also exposes the rocksdb.is-write-stopped and rocksdb.actual-delayed-write-rate properties, which should be graphed continuously.

L0 is the usual culprit because L0 files overlap. The L0-to-L1 compaction must merge all overlapping L0 files with the whole L1 key range it covers, and in the classic design that compaction is hard to parallelise. If it falls behind, every read slows down too, since each lookup checks every L0 file. This feedback loop (more L0 files, slower reads, slower compaction, more L0 files) is why the stall thresholds exist at all.

Mitigations the wiki recommends

The RocksDB wiki is explicit about remedies. If flushes are the bottleneck, increase max_background_jobs for more flush threads and raise max_write_buffer_number so more memtables can queue. If L0 file count or pending bytes are the bottleneck, add compaction threads, increase write_buffer_size to reduce write amplification, and consider min_write_buffer_number_to_merge. Raising the stall thresholds only postpones the problem, and the wiki says as much: it avoids stalls but does not fix the lag.

Three application-level tools are underused. Setting no_slowdown = true in WriteOptions makes a write return Status::Incomplete() rather than block, so the application can shed load or return a 503 instead of hanging threads. The low-priority write option marks bulk or background writes as the first to be delayed, protecting latency-critical traffic. And WriteBufferManager with allow_stall can cap total memtable memory across many column families or database instances, stalling all writers when the budget is exceeded.

Rate limiting and I/O smoothing

Compaction is bursty: a big merge can saturate a drive and starve foreground reads. RocksDB’s rate limiter (NewGenericRateLimiter) caps flush and compaction write bytes per second, with fairness between flush and compaction and an optional auto-tuned mode that adjusts the limit to observed demand. The idea is to trade a slightly larger compaction backlog for stable read latency. Setting the limit too low is itself a path to compaction debt, so size it above your steady-state compaction need, which you can compute from user write rate times write amplification.

Other smoothing tools include bytes_per_sync and wal_bytes_per_sync, which spread fsync work rather than issuing huge syncs, and compaction_readahead_size, which improves sequential read efficiency on spinning or network-attached disks. On cloud block storage with provisioned IOPS and throughput caps, burst credits can mask debt for hours and then hide the cliff; a stall that appears exactly when credits run out is a classic signature.

Tombstones, TTLs and space reclamation

A delete in an LSM tree is a write: it inserts a tombstone that shadows older versions. The tombstone and the data it hides are only physically removed when a compaction merges them together and the tombstone reaches the bottom level (or no older data can exist beneath it). Until then both consume space and, worse, iterators must skip over them. A scan across a range that has seen heavy deletes can touch millions of tombstones to return a handful of rows.

RocksDB provides tools to push reclamation along: CompactOnDeletionCollector marks files with a high density of tombstones for compaction, periodic_compaction_seconds forces old files to be rewritten, and DeleteRange with range tombstones makes bulk deletes cheap to write while keeping the same obligation to compact. Queue-like workloads that insert and delete in key order are the textbook trigger, which is why using an LSM tree as a message queue with keys in time order is a well-known anti-pattern.

Key-value separation and large values

If values are large, compaction spends most of its bandwidth moving bytes it never needs to sort. With integrated BlobDB, values above min_blob_size are written to blob files and only a small reference is stored in the SSTable, so compaction rewrites mostly keys. Garbage collection of blob files is driven by options such as enable_blob_garbage_collection and an age cutoff. The trade-off is one extra random read per lookup and added space amplification from blobs awaiting collection. As a rule of thumb, consider it only when values are many times larger than keys (hundreds of bytes to kilobytes and up) and the workload is write-heavy with moderate read concurrency.

LSM Tree vs B-Tree: Where the Bargain Pays Off

The honest comparison is not “LSM good, B-tree bad”. It is a trade of write cost against read cost and predictability. A B-tree such as InnoDB’s pays random page writes and page-split overhead but serves a point read with a handful of page accesses and a range scan with sequential leaf traversal. An LSM tree pays continuous background compaction and slower, more variable reads but absorbs ingest at sequential speeds.

Dimension	LSM tree (leveled)	LSM tree (tiered/universal)	B-tree
Write path	Sequential WAL plus memtable	Same	Random page writes plus WAL
Write amplification	High (often above 10)	Lower (single digits to low tens)	Moderate, driven by page size
Point read	Good with bloom filters	Slower, more runs	Excellent, stable
Range scan	Good, one run per level	Worse, many runs	Excellent
Space amplification	Low (about 1.1x ideal)	High (up to 2x headroom)	Moderate, fragmentation
Latency predictability	Compaction and stalls cause tails	Same, bursty large merges	Generally steadier
Compression	Strong, large immutable blocks	Strong	Weaker, page-level
Best fit	Write-heavy, SSD, compressible data	Very write-heavy, space to spare	Read-heavy, update-in-place OLTP

Two further points favour LSM engines. Immutable files compress better and enable cheap snapshots and backups (hard-link the files), and they make replication and point-in-time reads straightforward. Against that, LSM engines complicate transactional semantics at the storage layer: engines like RocksDB offer optimistic and pessimistic transaction libraries, but the isolation machinery is yours to configure. The absence of a global, fixed page structure also makes capacity prediction harder, which brings us back to the discipline of measuring amplification directly.

Trade-offs, Gotchas, and What Goes Wrong

The following failure modes account for most LSM incidents in production. Each has a visible signature and a first-line response.

Write stalls from underprovisioned compaction. Signature: periodic latency cliffs, L0 file count climbing, pending compaction bytes growing. First response: confirm the cause in the LOG, raise max_background_jobs, check disk bandwidth saturation, and ask whether write amplification is higher than planned. Longer-term, pick a policy with lower write amplification or add hardware.

Space blow-up during compaction. Leveled needs some temporary headroom; universal can need up to the size of the data being merged. A disk at 85 percent full is one big compaction away from an out-of-space event that corrupts nothing but halts the node. Alert on free space with headroom for your largest compaction, not on a fixed percentage.

Tombstone storms. Mass deletes, TTL expiry and queue-like access create scans that crawl through dead keys. Use range deletes, deletion-triggered compaction, and key designs that avoid scanning recently deleted ranges.

Read amplification creep. Reduced bloom filter bits, a block cache that is too small for the index and filter blocks, or L0 backlog all inflate reads gradually. Monitor rocksdb.bloom.filter.useful against rocksdb.bloom.filter.full.positive and the block cache hit ratio.

Cold-range starvation. With default priority, ranges that are never written may never be compacted, leaving old tombstones and stale versions in place. periodic_compaction_seconds and ttl address it at the cost of extra rewriting.

Misreading write amplification at the wrong layer. The LSM engine’s amplification is only one multiplier. Flash translation layers add their own (often 1.1x to 3x depending on over-provisioning and workload), file systems add journaling, and replicated systems multiply again by the replication factor. The end-to-end figure is the product, and it determines drive wear.

Cargo-cult configs. Copying a production options file from a different workload is a reliable way to inherit its stalls. Options such as write_buffer_size, target_file_size_base and max_bytes_for_level_base encode assumptions about key size, value size and write rate.

Practical Recommendations

Start from the workload, not the options file. Characterise write rate, key and value sizes, overwrite ratio, read mix (point vs scan), and delete patterns. Then choose compaction style: leveled when reads, scans and disk space matter most; universal when sustained writes dominate and you have headroom; FIFO for expiring telemetry.

Next, measure before tuning. Compute your actual write amplification from engine counters, and compare compaction throughput to ingest times amplification. If compaction is the bottleneck, no amount of option twiddling will beat adding threads, faster disks or a lower-amplification strategy. Change one parameter at a time, and run a realistic load long enough (hours, not minutes) to leave the memtable-absorbing honeymoon period and reach steady-state compaction.

A starting checklist, to be validated against your RocksDB version and workload:

Confirm level_compaction_dynamic_level_bytes is enabled (the wiki says it is the default since 8.4) so level sizes track real data volume.
Size write_buffer_size and max_write_buffer_number to your write burstiness and memory budget; consider a merge of two or more memtables for skewed keys.
Set max_background_jobs in line with CPU cores and disk parallelism; add max_subcompactions for large L0-to-L1 merges.
Use around 10 bits per key bloom (or Ribbon filter where CPU allows), and keep index and filter blocks in the block cache with high priority.
Enable compression by level: none or LZ4 for upper levels, Zstd at the bottom, where most data lives.
Put a rate limiter in place if compaction competes with foreground reads, sized above steady-state compaction need.
Graph write-stall indicators, pending compaction bytes, L0 file count, and the three amplification ratios, and alert on trends rather than on stall events.
Use no_slowdown and low-priority writes so applications fail fast or degrade gracefully instead of piling up.
Keep disk headroom of at least the size of your largest planned compaction.
Test failure and recovery: WAL replay time, backup restore time, and behaviour with a nearly full disk.

If you use a database built on one of these engines rather than embedding RocksDB directly, many of these knobs are exposed through the database’s own settings, with different names and sometimes different defaults. Check the vendor documentation rather than assuming RocksDB defaults apply.

Frequently Asked Questions

What is an LSM tree and why is it fast for writes?

An LSM tree buffers writes in an in-memory sorted structure called a memtable, logs them to a sequential write-ahead log for durability, and flushes them as immutable sorted files. Because the foreground work is only one sequential append and one memory insert, write throughput stays high regardless of database size. The cost is deferred: background compaction later merges files, rewrites data and consumes disk bandwidth, which is what write amplification measures.

What is write amplification in RocksDB and how do I measure it?

Write amplification is the bytes the engine writes to storage divided by the bytes the application wrote. It comes from the WAL, memtable flushes and, mostly, compaction rewriting data as it moves between levels. You can compute it from RocksDB statistics such as flush and compaction write byte tickers, or from the compaction stats printed in the LOG, divided by user bytes written. Measure at steady state, not on a fresh database.

Is leveled or tiered compaction better?

Neither wins everywhere. Leveled compaction gives low space amplification and good read performance, at the price of high write amplification. Tiered and universal compaction cut write amplification but increase read amplification and need more free disk space for merges. Choose leveled for read-heavy or space-constrained systems and tiered for write-heavy ones with headroom. Hybrids such as lazy leveling and Cassandra’s unified compaction strategy sit between the two.

What causes RocksDB write stalls?

Three conditions, per the RocksDB wiki: too many immutable memtables waiting to flush, too many level-0 files, or too many estimated pending compaction bytes. Each has a slowdown threshold and a stop threshold. They signal that compaction or flush cannot keep pace with writes. Fixes include more background jobs, larger or more write buffers, a lower-amplification compaction style, or shedding load with no_slowdown writes.

How does a bloom filter speed up LSM reads?

A bloom filter attached to each SSTable answers whether a key is definitely absent or possibly present. A definite “absent” lets the engine skip the file without any disk read. At about 10 bits per key the false-positive rate is roughly 1 percent, so most files in a lookup are skipped. Bloom filters help point lookups and prefix scans but not arbitrary range scans.

When should I choose a B-tree instead of an LSM tree?

Prefer a B-tree when reads and range scans dominate, when predictable latency matters more than peak ingest, or when updates are in place and the dataset fits well in memory. Choose an LSM tree when writes are heavy, data compresses well, storage is SSD-based and you can tolerate compaction-related tail latency. Many systems offer both engines, so benchmark with your own access pattern before committing.

LSM-Trees and RocksDB: Compaction, Write Amplification and Tuning (2026)