We added twenty read replicas. Why did the primary get slower?

A replica is a reader, but also a consumer of writes

A read replica does not receive updated tables by magic. The primary generates WAL for committed changes, and each replica needs a suitable stream of those records. The replica then replays the changes to maintain its own database state.

Imagine the primary generates 40 MB of WAL each second. One direct copy is roughly 40 MB/s before protocol overhead and retransmissions. With twenty independently streamed replicas, a naive full fanout can require about 800 MB/s of outbound payload from the primary. That is approximately 6.4 Gbit/s, before counting its ordinary application traffic. Wire rates vary with topology and compression. This is a hypothetical estimate.

More replicas can reduce query CPU on the primary while consuming its network bandwidth, WAL sender resources, and operational attention. If network transfer queues grow, replicas become less current. The application may then route freshness-sensitive reads back to the primary, returning some of the original load when it can least afford it.

That feedback is not a property of every topology. The interview task is to measure which resource actually moved after the scale-out.

Find the hidden cost of a lagging replica

A replica that falls behind does not just miss recent data. The primary may have to retain old WAL so the replica can catch up. In PostgreSQL, replication slots can prevent removal of WAL needed by a consumer. Without an appropriate bound, a long-stalled slot can retain enough data to fill the primary's WAL disk. PostgreSQL documents max_slot_wal_keep_size as one control, with the important warning that a replica may lose the WAL it needs when a limit is enforced. PostgreSQL replication settings.

At the illustrative 40 MB/s generation rate, retaining twenty-four hours of continuous WAL would represent about 3.46 TB of raw WAL before other overhead. That amount is a calculation from the assumed rate, not a forecast for every database. The primary retains a shared WAL range back to the oldest required position; twenty slots do not imply twenty separate copies of that same retained range. If a slot pins the oldest position, adding disk only buys time. The underlying replication consumer still needs a plan to recover or be rebuilt.

There is another possible interaction: hot standby feedback can postpone removal of row versions needed by standby queries, increasing table or index bloat on the primary in some workloads. The tradeoff is between canceling conflicting standby queries and retaining dead tuples longer. PostgreSQL standby documentation.

I would inspect the primary's outgoing WAL rate, WAL sender activity, retained WAL bytes, slot restart positions, network saturation, and primary disk latency. On the replicas I would compare receive, flush, and replay progress, plus query conflicts and replay throughput. A lagging replica may have a fast network and a slow apply stage. Adding bandwidth will not fix that.

Are twenty replicas the right read architecture?

It depends on why the reads are expensive. If the primary is serving many ordinary point reads and most clients tolerate a little staleness, several regional replicas may be useful. But if twenty replicas each run a full-table aggregation, replication has multiplied storage, replay work, and infrastructure cost without changing the shape of the query.

For analytics, I would consider an appropriate derived reporting store or materialized view refreshed through a controlled pipeline. A precomputed daily aggregate can reduce repeated scans substantially. It introduces freshness and repair contracts, but it may use fewer resources than cloning twenty transactional databases to run the same large query.

Before proposing a warehouse, also check indexing and query plans. A bad index choice replicated twenty times remains a bad query. Query optimization might remove more load than another server. Conversely, one CPU-heavy report that genuinely needs its own isolation may justify a replica even at modest traffic.

Separate replicating the full database from materializing particular answers. They are different investments.

Could cascading replication fix the primary fanout?

Potentially. A few first-level replicas can receive WAL directly from the primary, and downstream replicas can stream from those intermediates. PostgreSQL supports cascading replication. This can reduce direct sender connections and network fanout from the primary, but adds another dependency and may introduce extra lag or recovery complexity. PostgreSQL streaming replication.

A cascade should have a placement policy. If many replicas depend on one intermediate, its failure can interrupt updates for an entire branch. Cascading also does not reduce the amount of log processing and storage across the entire fleet. It relocates where the network fanout occurs.

If the original requirement is “users must always see the latest committed value,” adding asynchronous replicas will not by itself satisfy it. If reads can tolerate staleness, the acceptable bound must still be measured. And if the team chooses synchronous replicas, adding more required acknowledgment participants can increase commit latency or reduce availability depending on quorum and policy.

The interview verdict should be evidence-based. Replicas are not free readers. They are continuous subscribers to the write history, with their own disk, apply, retention, and failure demands. Scale reads by measuring the whole data path, not just counting database connections.