Learning series · 6 episodes · Senior / Staff / Principal
The Config Changed. Who Told the Fleet?
Six connected engineering decisions about coordination on the request path, stale discovery, config rollouts, feature flags, routing safety and controller convergence.
Start series →Follow the decisions in order
- Part 1 of 6
Senior
ZooKeeper is healthy again. Why is every API request still waiting for it?Move ordinary configuration reads into validated atomic local snapshots, with safe bootstrap, watch recovery and differentiated staleness policies.
- Part 2 of 6
Staff
The dead instance was removed. Why do clients keep calling it?Distinguish updated service registry membership from per-client caches, existing connection pools and graceful versus unexpected backend loss.
- Part 3 of 6
Principal
One configuration field changed. How do you roll it out to 1,000 services?Deliver versioned, semantically validated and compatible configuration to large fleets through measurable staged rollout and tested rollback.
- Part 4 of 6
Staff
The feature flag says off. Why are some customers still seeing the feature?Design deterministic local feature-flag evaluation, context propagation, version observability and critical kill-switch limits.
- Part 5 of 6
Principal
The routing config passed every check. Why did one cell receive 70% of traffic?Reject routing configurations that pass parsing but violate weighted capacity, state ownership, residency or safe rollback conditions.
- Part 6 of 6
Principal
One resource changed. Why did thousands of controllers start hammering the API?Use list/watch, coalesced rate-limited reconcile queues, controlled retries and idempotent external operations to recover from watch and API-server interruptions.