ZooKeeper is healthy again. Why is every API request still waiting for it?

Interview question. A company stores feature configuration and backend addresses in ZooKeeper. Hundreds of services call getData or getChildren before processing every API request. The coordination cluster becomes overloaded and application latency collapses. Design the data flow you would change, and explain exactly how services recover after a missed update.

A coordination service is good at answering a question such as, "Which configuration revision has been approved?" That does not mean it should answer the same question 50,000 times each second for individual customer requests. Imagine checkout needs a fraud-rule threshold and the identity of a tax endpoint. The team has placed both in ZooKeeper because updates there are coordinated. It feels safer than copying configuration into every process. Now every checkout inherits an extra network request, an external availability dependency, and a connection to a system designed for coordination rather than application traffic.

Consider 400 instances receiving 200 requests per second each. If each request makes two coordination reads, the fleet produces 400 × 200 × 2 = 160,000 reads per second before accounting for retries. This is a hypothetical workload, not a published ZooKeeper capacity limit. Even if the coordination cluster handles that many reads, the design couples every customer's latency to the health of the central service. Adding application instances actually increases pressure on the shared dependency.

The fix begins by identifying which values are configuration and which are decisions requiring current authoritative state. A shipping cutoff, risk-model version, or service endpoint list may be consumed from a versioned local snapshot. A single-use authorization or globally unique allocation cannot become safe merely because we cached it. Do not replace a true coordination operation with a stale local guess. First classify keys by correctness requirements, expected update rate, maximum acceptable staleness, and failure behavior.

For ordinary configuration, make requests read memory. A per-process snapshot can be stored as one immutable object, validated completely and exchanged atomically rather than mutating several settings independently. The service initializes from an approved snapshot before readiness. A background controller watches for changes, fetches the complete configuration version, validates its schema and cross-field dependencies, then swaps the active pointer. A file-based agent or sidecar can also distribute snapshots, but adds a local process boundary and its own restart considerations. The critical rule is that a customer request never synchronously waits for ZooKeeper to check whether the value has changed.

ZooKeeper watches deserve careful treatment. In the traditional ZooKeeper API a watch is a one-time notification, not a durable subscription that guarantees every intermediate change. A watched node can change from revision 7 to 8 to 9 before a disconnected watcher rearms. The watcher should treat the notification as a reason to read the current value, not as the authoritative new state itself. It needs a reconnect path and a fresh read that reestablishes observation. The watch also may fire for a deletion, not just a modification. Holding revision 7 forever because no notification arrived would silently preserve a retired rule.

A robust implementation reads a complete snapshot and establishes a watch according to the API's ordering semantics. When notifications occur, it schedules a coalesced refresh. It records the last successfully applied version, hash, validation result, time of last contact, and any pending change. On reconnect, it compares local state with the current authoritative snapshot, rather than assuming that no events were missed. If the store exposes a monotonic revision, reject an older snapshot for that version domain. Be careful: different independent keys may not share a useful atomic revision unless the configuration is published as one document or through a consistent manifest.

What happens during a one-hour outage? A healthy process should keep reading the last known approved configuration when the product contract permits that staleness. This is not the same as unconditionally failing open. A fraud rule that must immediately block compromised accounts, or a newly revoked credential, may need a short validity lease and a deliberate fail-closed policy. Conversely, stopping all product-page reads because a theme color cannot update is unnecessary. Write a per-field or per-bundle policy describing whether an expired snapshot is usable, degraded, or prohibited. Avoid returning success when the system cannot enforce a required authorization decision.

Now consider a new pod started while the coordination store is unavailable. Existing pods can continue with a validated local snapshot, but a new pod has no such state. It should not become ready based on an empty map or copied defaults that accidentally turn off controls. Distribute a signed or integrity-checked last-approved bundle through a separate, deliberately provisioned bootstrap channel if startup during outages is a requirement. If the bootstrap bundle is missing or too old for a security-critical rule, that pod remains not-ready. Static stability requires designing the bootstrap path ahead of the incident.

After coordination recovers, resist letting all 400 instances synchronously perform a full read. Use jitter, capped refresh concurrency, hierarchical dissemination or a fanout agent. Suppose each of 400 instances fetches a 2 MB bundle simultaneously from the artifact distributor: that tier would transmit roughly 800 MB just for one round of updates, ignoring retries and overhead. Keep only a small approved-version pointer or manifest in ZooKeeper; a single 2 MB znode exceeds its normal data-size limit, and large bundles belong in a bulk artifact store. Small snapshots make this easier, but the essential decision is separating a slow configuration path from fast application requests. Expose metrics for revision lag and count of instances per active version, not just whether ZooKeeper has a leader.

The interview answer should end with an exercise in failure behavior. Disconnect ZooKeeper for an hour, change the approved config while clients are disconnected, restart half the app fleet, and then restore coordination. Existing safe requests should keep completing within their normal latency budget, revoked operations should follow their stricter policy, new instances should enter service only with valid bootstrap state, and every permitted instance should eventually converge to the latest approved version. The architecture is successful when the coordination system coordinates change without becoming a mandatory participant in every customer transaction.