Interview question. A service backed by an LSM engine sustains its target write rate during a short load test. After about an hour, p99 write latency jumps and requests begin timing out. Application traffic and code have not changed. Some writes appear to be intentionally delayed inside the storage engine. Diagnose the failure and propose changes that improve steady-state behavior without trading away the data's durability contract.
Assume one RocksDB-style node for the first diagnosis. Compaction and flushing use the same finite storage and CPU resources as foreground work. Keep the first diagnosis focused on local compaction and flushing; network retries and API gateway queues are separate sources of pressure.
Think first
A write-heavy LSM service looks fast during the first 20 minutes of a load test. After 90 minutes, L0 files grow rapidly, the engine logs write stalls, and user latency climbs despite constant inbound QPS. Construct a diagnosis and a steady-state acceptance test. Would you raise the L0 stop trigger?
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
Why the initial benchmark was convincing
The load test started with an empty database. Every new update entered the WAL and a memtable. The memtables flushed into fresh L0 files. For a while, the storage device had plenty of free space and the background compaction backlog was small.
The benchmark measured how fast the engine accepted writes, not whether it could complete the resulting background work.
That is the first question I would ask: what was still left for the engine to do when the benchmark stopped?
RocksDB explicitly slows or stops writes when flushing or compaction cannot keep up. This is protection against unbounded read amplification, space growth, and eventually exhausted disk capacity, not automatically evidence of a broken storage engine. Its documentation describes separate triggers for too many unflushed memtables, too many L0 files, and too many bytes pending compaction. 1
The background queue you did not size
Think of each new SST file as a future obligation. Recent L0 files can overlap in key range, so leaving them untouched can increase the work a read performs. Compaction reads and merges files, writes replacement files, and safely changes the live file set. The process consumes disk reads, disk writes, CPU, and temporary free space.
Incoming flush files accumulating faster than compaction can retire them, then reaching a write-stall threshold
Here is an intentionally simplified work-rate model. Suppose incoming traffic creates an estimated 120 MiB per second of required future compaction work, measured in some consistent work-unit definition. The engine can retire only 90 MiB of that work per second under current competing load.
Compaction work arriving 120 MiB/s
Compaction work completed 90 MiB/s
Net accumulation 30 MiB/s
After one hour 108,000 MiB, about 105 GiB
These are illustrative rates of required compaction work, not RocksDB pending-byte metrics. Actual compaction amplification and scheduling are more complex. The lesson is the sustained mismatch between work arriving and work completed.
Once too many L0 files or pending bytes accumulate, the engine can slow incoming writes to let compaction catch up. Removing the slowdown threshold without increasing sustainable background capacity may produce better short tests and worse eventual failure.
Diagnose before increasing threads
I would inspect the RocksDB logs for explicit stall reasons and the engine's per-level statistics. Which condition actually triggered the delay? Are immutable memtables accumulating because flush workers cannot finish? Are L0 files growing because L0-to-L1 compaction is stuck? Is pending compaction work increasing even while compaction threads are busy? Does another column family cause a database-wide stall?
Then inspect system-level contention. Are disks saturated on reads, writes, or both? Is compaction CPU-limited? Are the SST files spread over a storage device whose sustained write performance falls under garbage collection or low free-space conditions? Are long scans or snapshots preventing cleanup of old versions? Is there enough temporary disk headroom to produce replacement files safely?
If foreground writes and compaction together need more sustained I/O than the device supplies, extra compaction threads cannot create bandwidth. They may instead slow foreground reads.
Fix the limiting stage, not the visible symptom
If flushes are the bottleneck, reserve sufficient background capacity for flushing and consider buffer sizing so short bursts can be absorbed without filling every memtable. If L0-to-L1 is the limit, investigate file overlap, compaction configuration, and whether subcompaction can help this key-range distribution. RocksDB documents this bottleneck and its supported subcompaction mechanism. 2
If the device is saturated, reduce rewrite work, isolate I/O, improve storage capability, or limit admitted writes. Leveled and more tiered compaction styles trade write, read, and space amplification differently. There is no free setting that minimizes all three. 3
I would also check the data model. Rewriting huge values when only a small field changes can increase byte movement. A poorly chosen key distribution may prevent parallel background work. Short-lived test data can make the average obsolete-version rate unlike production.
A larger write buffer can postpone a stall, but cannot eliminate steady-state maintenance.
Prove the new design has reached steady state
For acceptance, I would use a long enough test to fill multiple levels and enter a representative update and delete cycle. Compare time series of write latency, completed useful writes, L0 file count, immutable memtables, pending compaction work, disk free space, and background bytes read and written. The backlog should remain bounded rather than increasing until the test finishes.
Also test a failed disk, restart, and recovery under the configured durability policy. Optimizing compaction must not cause the engine to acknowledge transactions before the promised durable boundary.
The strongest interview answer explains why a write-optimized engine intentionally refuses more work. It distinguishes a transient burst from a permanently unsustainable service rate, identifies the exact stall gate, and proposes a measured capacity or amplification improvement rather than turning off safety limits.
A stable engine still needs a data layout that fits the workload. Range scans, point reads and updates to secondary indexes impose different costs. Evaluate those query patterns separately from whether compaction can sustainably keep up.
Go deeper
Different triggers imply different repairs
Too many immutable memtables points toward flushing. Too many L0 files often indicates the L0-to-L1 path is insufficient. Too many estimated pending compaction bytes suggests deeper work is falling behind. Each can lead to slowed writes, but the corrective action and limit differ. RocksDB's stall descriptions are per configured column family, while a stall may affect writes to the broader database. 1
A critical operational trap is tuning only the initial workload. An empty LSM tree has not yet accumulated the deepest levels, nor has it paid steady-state rewrite costs. Use populated and aging datasets, including realistic overwritten-key distributions and deletions. Keep enough free disk space for compaction output. A nearly full disk can prevent recovery work precisely when the store most needs it.
Increasing compaction concurrency only helps when work can be divided and the underlying resources are not already saturated. L0 overlap can constrain parallelism, and a compaction task can temporarily compete with reads for cache and I/O. The correct intervention might be a better level-size policy, a faster device, fewer bytes written per update, or tighter admission rather than more threads.
One useful metric is write amplification, expressed with an explicit definition such as physical storage bytes written divided by logical user bytes written over the same interval. If a foreground 1 GiB of new logical data causes 8 GiB of total device writes during steady state, the system must provision for that work. This example is not an asserted RocksDB default. Measure it on the target workload.
Check your reasoning
Ask whether the workload started with an empty tree and whether compaction had reached steady state.
Inspect actual stall reason including immutable memtables, L0 file count, or pending compaction bytes.
Compare incoming background work with sustainable compaction completion and disk/CPU/free-space budgets.
Select an intervention that improves the limiting stage or reduces work rather than hiding it behind thresholds.
Verify a bounded backlog under representative updates, deletes, snapshots, and recovery requirements.
Common mistake: Assuming the request path must be at fault because it suddenly slows, or removing stall limits while compaction debt continues to grow.
Read the answer explanation
Delay the stall but not fix the mismatch between incoming and completed background work. — Threshold changes can move the point of visible throttling but cannot create the physical work rate needed to stabilize the backlog.
Apply it elsewhere
A stream-processing state store runs RocksDB on every worker. During a checkpoint surge, compaction stalls increase. Where would you instrument first?
Compare your approach
Measure competing checkpoint I/O, flush and compaction throughput, L0 file counts and actual stall triggers. Consider workload coordination or resource isolation before blindly adding compaction threads.
One more constraint
The disk is at 60 percent utilization while L0 files continue growing. What resource or scheduling constraints might still block compaction?
Quick revision
In an LSM, foreground writes create sorted files and deferred compaction obligations. A short benchmark can look excellent before those obligations accumulate. When flush or compaction falls behind, immutable memtables, L0 files, or pending compaction bytes increase. RocksDB may deliberately slow or stop writes to prevent read amplification and disk usage from becoming unbounded. I would inspect the explicit stall reason, per-level file counts, compaction throughput, disk and CPU saturation, and free-space headroom. Then fix the limiting flush or compaction stage, reduce amplification, add sustainable capacity, or bound write admission. I would reject a fix that merely raises thresholds while debt grows forever. A successful test must reach steady state and retain the promised crash-recovery behavior.
