The interview question. Your order database has one primary and three read replicas. A transaction commits successfully, and monitoring shows that a follower has already received the corresponding WAL. A read routed to that follower still returns the previous order status. Is the replication broken? Explain the distinct stages of replication and design a reliable way to decide when a follower may serve a read that must include the write.
Assume a PostgreSQL-style physical streaming replica for the concrete implementation. The primary has acknowledged a transaction. There is no regional promotion or competing primary in this question. We are examining the distance between a replicated record and a query that can see its effect.
Think first
The primary acknowledged a commit at L120. A read replica has received and flushed WAL through L125 but replayed only through L117. Can it safely answer a read that must contain L120? Explain what each position means, what you would measure, and how the router responds before a 150 ms deadline.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
Follow the commit record, not just the network packet
The database records a change and its transaction outcome in the write-ahead log. A replica receives log bytes, writes them, may flush them to durable storage, and then replays the records into its local database state. Those are not one indivisible operation. Bytes can arrive while replay remains behind.
Imagine o481 moving from ACCEPTED to RESERVED. The primary commits at log position L120. A follower has received through L125, and has flushed through L125, but its replay position is L117. A query on the follower may still see ACCEPTED. Its durable log contains the future state, but query visibility has not reached it.
A write can be received and flushed on a follower before replay makes it visible to a query
This is useful language for an interview. Received, flushed, and applied are different observations. So is acknowledged, because acknowledgment depends on what the primary promised. A locally acknowledged asynchronous write can be less protected than a remotely flushed one. A remote flush can protect durability without guaranteeing immediate follower visibility. PostgreSQL exposes these differences through its replication statistics and synchronous commit options. PostgreSQL WAL configuration.
There is also an important distinction between two families of replication. A consensus-replicated state machine often distinguishes a committed log index from the last applied index. A physical PostgreSQL standby talks instead about received, flushed, and replayed WAL positions. The mechanisms are related but not identical. I would never infer that a follower can serve a current read merely because a coordinator uses the word committed.
Decide what the read needs to promise
Not every read should wait for the most recent write. A public product catalogue may accept a short delay. An order confirmation screen generally should not show the status it displayed before a successful update.
For the second case, I would return a replication position or a suitable monotonic version along with the successful write. A subsequent read carries its minimum required position. The routing layer selects a replica that has replayed at least that position. The read must also acquire its snapshot after that replay barrier; an already-open Repeatable Read transaction keeps its older snapshot. If none qualifies before the request deadline, it can read from the primary, wait within the allowed budget, or return a clearly identified temporary-unavailable outcome. Which option is permitted is part of the API contract.
The minimum position must be issued by a trusted writer, not an arbitrary timestamp supplied by the browser. Under one stable replication history, comparing positions is meaningful. After topology changes or promotion, position alone may not have the same meaning. The server needs to validate a timeline or equivalent authority identity. Here topology stays stable; regional promotion would require separately proving which write history remains authoritative.
A tempting implementation waits until the follower's network receive pointer has passed the token. That is the original bug translated into code. The criterion must correspond to visible state. On a transactional WAL standby, the relevant transaction's commit must have been replayed before the promise is met.
Why not make every commit wait for replay?
PostgreSQL provides a setting named remote_apply when configured with synchronous standbys. Under that policy, the primary waits until the selected standby has applied the transaction and it is visible to standby queries. remote_write and the ordinary synchronous on setting have different acknowledgment boundaries. These distinctions are documented by PostgreSQL, not universal configuration names across databases. Synchronous commit options.
A stronger commit policy can simplify some read paths, but it makes write latency depend on remote replay, including replay contention. It can also make commits wait when a required standby is unavailable. Paying that cost for every write may not be necessary when only a minority of reads require immediate visibility.
As an alternative, write to the primary and perform the immediately following critical read there. It may concentrate load. Position-gated follower reads provide more flexibility when replay keeps up. Neither choice should be made without the read-frequency and latency requirements.
Diagnose the difference in production
I would compare the primary's WAL send position with each replica's receive, flush, and replay progress. PostgreSQL's pg_stat_replication reports write, flush, and replay lag observations. It is dangerous to treat one lag duration as an exact timer predicting when a particular future write becomes visible. The reported times can remain briefly after catch-up and later become null. Replication monitoring.
If receive falls behind, investigate bandwidth, backpressure, and the sender. If flush falls behind, investigate replica storage. If replay falls behind while flush is current, investigate apply activity, conflicting queries, CPU, and storage pressure. These need different responses.
Suppose a 200 ms read deadline expires because the follower cannot reach the required replay position. “Try another stale follower” is not progress. The router needs a bounded fallback or a refusal that preserves the stated consistency contract.
The key interview distinction is precise: durable on the replica is not the same as visible through a read served by that replica. Once that is clear, a harder application question appears. How should the service maintain the visibility guarantee for a customer moving between devices that do not share a connection or process?
Go deeper
Further technical depth
Progress pointers. PostgreSQL can expose sent_lsn, write_lsn, flush_lsn, and replay_lsn for streaming replication. Avoid treating them as interchangeable. In particular, a standby can have bytes safely stored without applying the commit needed by a query. Depending on where monitoring runs, use primary and standby views and account for monitoring delay.
Query conflicts. Long-running reads on a hot standby may conflict with recovery replay. Some settings delay replay or cancel conflicting reads. If the application depends on a small freshness bound, that recovery policy is part of the product's latency story, not a tuning detail hidden from reviewers. PostgreSQL hot standby.
Tokens are not timestamps. A replay or commit position identifies progress in a particular ordered history. Comparing positions from different histories after failover can be invalid. Include a generation or authority identifier when issuing tokens, and provide a recovery policy when it no longer maps to the active history.
Do not overstate remote_apply. Waiting for a selected synchronous standby to apply does not make arbitrary other async followers current, and it does not guarantee all application caches reflect the write. The API still chooses where it reads.
Check your reasoning
Separate WAL receive, flush and apply or replay, rather than treating any received bytes as query-visible.
State that the exact primary acknowledgment guarantee depends on the replication and synchronous-commit policy.
Require replay through the relevant committed transaction before allowing a version-gated follower read.
Specify bounded wait, eligible replica, primary fallback or honest unavailability if the deadline is reached.
Distinguish one replication timeline from a changed authority after failover.
Common mistake: Calling remote durability and follower visibility the same thing, or checking received position to authorize a fresh read.
Read the answer explanation
Wait for replay through the commit, route to an eligible authority, or return unavailable. — The bytes are present but the corresponding commit has not been replayed into visible follower state.
Apply it elsewhere
A Raft-backed metadata store has commit index 700 and applied index 693. Can its follower locally answer a query requiring committed change 698?
Compare your approach
Not solely from the commit index. The follower must apply the needed state and satisfy the specific read protocol, potentially using a leader or read barrier.
One more constraint
Why would remote_apply make write latency worse even if the network latency has not changed?
Quick revision
A replica can receive and durably flush a transaction's WAL before replay makes that transaction visible to queries. A current-read contract must check the appropriate apply or replay progress, not just delivery of log bytes. Use primary reads, position-gated replicas, or a stronger commit policy according to the latency and availability requirements. PostgreSQL remote_apply waits for selected synchronous standbys to apply, but it does not make every other replica current. The key question is which exact event establishes the guarantee the client was promised.
