Skip to content

Two of three replicas agreed. Why did a later read return older data?

The copies disagree · Part 3 of 5 · 6 min read

Read the worked explanation ↓

The interview question. A key-value store has three replicas, with write quorum W = 2 and read quorum R = 2. Engineers say that W + R > N, so a read must return the latest value. During an incident, one read returned version 5 and the next returned version 4. Explain a possible execution. State precisely what quorum overlap provides and what additional behavior the store needs before it can promise monotonic or linearizable reads.

Use a Dynamo-style, leaderless replicated key for this exercise. Assume a stable replica set A, B, C, a version-aware coordinator, and no writes from another key. Do not import the safety rules of a Raft log into a system that has not implemented Raft.

Think first

Leaderless replication uses N=3, W=2 and R=2. Version 5 reached A alone before the writer timed out. A read from A+B returns v5 without blocking repair, then a read from B+C returns v4. Explain why the inequality did not prevent this and which guarantee you can honestly offer.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

What the arithmetic really proves

With N = 3, any two sets of two replicas must intersect. For example, a successful write acknowledged by A and B overlaps a later read from B and C at B. If version 5 is the latest completed, acknowledged write and the read compares valid versions correctly, B can reveal that version even if C has not received it yet.

This is a valuable building block. It is not a complete protocol. The intersection does not settle partial writes, concurrent updates, or read repair.

The following incident uses a failed or uncertain write. That distinction matters more than the inequality.

A write that reaches only A can be seen by read A+B, then missed by read B+C when read repair is not blocking
A write that reaches only A can be seen by read A+B, then missed by read B+C when read repair is not blocking

Initially all three replicas store version 4. A client writes version 5, but only A applies it before the coordinator times out. The client never receives a successful W = 2 acknowledgment. We cannot say the write completed under that contract.

Now read one contacts A and B. A returns version 5 and B returns version 4. The coordinator selects version 5 as newer and returns it. Assume this demonstration protocol does not make the read wait for a completed repair to B. A later read contacts B and C, both still at version 4. It returns version 4.

Two quorum reads, two different outcomes. The arithmetic was not violated. The only version 5 write never achieved the write quorum, and a read exposed its partial state without establishing a new durability or repair boundary. What felt like a simple freshness issue is now a question about the exact obligations of the read algorithm.

Monotonic reads and read repair

One response is to make the first read repair the inconsistent replicas before returning. The coordinator can propagate the chosen value until an appropriate acknowledgment boundary is reached. Under suitable assumptions, this prevents a later quorum read from losing a value that an earlier completed quorum read exposed.

Apache Cassandra documents precisely this tradeoff. Its default blocking read repair supports monotonic quorum reads in cases involving partially completed writes. Configuring read_repair = NONE favors partition-level write atomicity but may lose that monotonic quorum behavior. This is a documented Cassandra implementation choice, not a promise supplied by the inequality itself. Cassandra read repair.

Why can read repair hurt write atomicity? If a mutation changed several related rows and a query repairs only the rows it examined, another reader might observe a subset of that mutation. The lesson is not to switch repair off automatically. The question is which invariant the application needs across its read and write shapes.

You can also preserve read progress in a client session token, much as in the previous lesson. But that protects clients carrying the token. It does not automatically make arbitrary other readers monotonic, and it does not turn concurrent writes into one globally ordered history.

What if two writers are active?

Suppose two clients concurrently update the same account preference. The store must resolve two candidate values. Last-write-wins by wall-clock timestamp is one possible policy, but clock skew can discard a value that was submitted later in real time. Vector version metadata may reveal concurrency without deciding the business winner. An application merge may preserve both changes when the data type allows it.

Even if every successful read overlaps every successful write, that does not establish a single real-time order for concurrent operations or an atomic compare-and-swap. Quorum overlap is a membership property. Linearizability is a history property. The latter requires a protocol that makes operations appear to take effect at one point consistent with real-time precedence.

If the application needs to decrement the last available seat with no overselling, using R = W = 2 and a blind read-modify-write is not automatically safe. Two writers may read the same old value and race. A consensus-backed conditional update, a single authoritative owner, or a suitable linearizable transaction is required for that invariant.

Failure assumptions change the answer again

The quorum calculation assumes the same defined replica set for the write and read. A sloppy quorum may accept substitutes outside the normal owners. If the read later contacts normal owners that never received the hinted write, the simple intersection claim may not hold in the way the application expects.

Replica reconfiguration also changes membership. If the system changes owners without preserving an overlapping committed history, quorum math for one membership epoch cannot be casually applied to another. Repair is not identical to safe ownership transfer.

When interviewing, begin by asking what “latest” means: latest attempted value, latest acknowledged value, latest causally dependent value, or latest operation in real-time order. Then identify how that guarantee survives partial writes, conflicting versions, repair, and membership changes.

For the next failure boundary, read why a deleted record can return after tombstone expiry and repair. That existing question explains deletion evidence before this series continues with the cost of adding read replicas.

Go deeper

Further technical depth

Quorum overlap example. With RF three, a successful W2 write on A and B intersects any later R2 read. If a reader selects the newest correct version among responses and there are no interfering writes, that is a useful freshness condition for completed writes. It says nothing on its own about atomic multi-row updates.

Read repair is not always cheap. Repair work can add foreground latency and extra write traffic. The Cassandra documentation explicitly discusses the relationship between blocking repair, monotonically advancing quorum reads, and partition-level write atomicity. Treat these as configurable tradeoffs rather than universal claims.

Partial operations remain visible. A write may return a timeout but leave versions on some replicas. A store needs an explicit rule for whether a read can reveal these and what, if anything, must be stabilized before returning.

Sloppy quorum. Dynamo-style hinted handoff can improve write availability by temporarily using fallback nodes. An R + W > N argument based only on normal owners must be reexamined when participants change.

Check your reasoning

  • Demonstrate intersection for completed W2 writes in a stable three-replica membership.
  • Separate failed or uncertain partial writes from acknowledged successful writes.
  • Describe read repair or session constraints needed to stop later quorum reads from moving backward.
  • Explain why concurrent writers need defined conflict resolution or stronger conditional-write coordination.
  • Avoid asserting that W+R>N alone establishes linearizability or works unchanged under sloppy quorum.

Common mistake: Treating the arithmetic as a linearizability proof while ignoring a partially visible write and the read-repair protocol.

What did N=3, R=W=2 guarantee in the example?

Read the answer explanation

Overlap between any successful R2 and W2 replica sets in the assumed static membership. — Intersection is a property of chosen replica sets. It does not settle partial writes, read repair or concurrent ordering.

Apply it elsewhere

A replicated feature flag can be enabled concurrently by two regions. What does quorum intersection not tell you about the winning value?

Compare your approach

It does not define merge or conflict resolution. Specify an authority, an ordering protocol or a data-type-specific merge contract.

One more constraint

How can blocking read repair protect monotonic quorum reads while affecting multi-row write atomicity?

Quick revision

R + W > N guarantees intersection of the selected replica sets under the assumed membership model. It helps a later read encounter an acknowledged write, but does not by itself guarantee linearizability, monotonic reads, or protection from concurrent updates. A partially applied write can be observed by one quorum read and missed by another if no suitable read repair or session rule stabilizes it. Specify the version-resolution protocol, repair boundary, and membership assumptions before promising correctness.

References