One resource changed. Why did thousands of controllers start hammering the API?

Interview question. A Kubernetes-like control plane has thousands of controllers that observe desired state and reconcile external resources. After an API-server restart, controllers simultaneously list objects, create work and retry failed updates. The API becomes saturated, resources oscillate between states, and ordinary operator commands time out. Design a stable reconciliation architecture and recovery policy.

A controller is a loop that asks two questions: what should the system look like, and what does it look like now? If the answers differ, it takes a bounded action to reduce the difference. This is fundamentally different from a deployment script that runs once and assumes every command succeeded. It is also different from a stream processor that must preserve every intermediate event. For many controllers, the important property is convergence toward the latest desired state, not replaying each change as an independent job.

Imagine 3,000 controllers reconnecting after a control-plane interruption. Each manages an average of 400 objects and performs a full list plus two follow-up API calls per object. The simple upper-bound illustration is 3,000 × 400 × 2 = 2.4 million follow-up calls, excluding the initial lists and any retries. Many controllers actually share informers or watch caches, so this is a warning about an inefficient design, not a claim that Kubernetes normally behaves this way. The restart storm consumes the very API capacity needed for recovery.

The key architectural unit is a workqueue containing an object identity such as namespace/name, not a queue of every raw change notification. If an object changes fifteen times before its worker processes it, the controller usually needs to reconcile the latest state once. The queue can deduplicate pending keys. When a worker dequeues one key, it gets the latest object from a local informer cache and compares its desired configuration with observed reality. Events trigger reconciliation promptly, but periodic resyncs or other repair paths help recover from missed signals. Coalescing is correct only if the reconciliation function is level-driven and can reconstruct what to do from current state.

This suggests a safer state machine. The controller reads a desired resource revision, inspects external state, computes a minimal action, and applies it with an idempotent API. The controller should then observe the result before declaring success. If the action has a non-idempotent side effect, attach an operation identity and manage partial completion explicitly. For example, provisioning a new volume must not create a second billable disk every time the controller retries after a timeout. Record the external provider's resource ID or idempotency key in durable status.

Watches complicate recovery. The Kubernetes API exposes resource versions, but it does not retain history forever. A watch from a resource version that has aged out may produce 410 Gone. The correct recovery is to re-list current state and start watching from an appropriate returned version, not to keep retrying the impossible old revision. In etcd, a watch on compacted history also fails. A controller that assumes its watch stream is a permanent complete event log will drift or loop after compaction. Standard informer/list-watch libraries exist partly to encapsulate this resynchronization contract.

Next constrain load. Use a bounded number of concurrent reconciles, a token bucket or equivalent limiter for outgoing API calls, and per-object exponential backoff with jitter for transient errors. A 429 from the API server is not an invitation to retry immediately on every controller. Honor any protocol-defined retry guidance, while ensuring deadlines and cancellation make sense. Apply separate caps for high-cost provider operations and low-cost metadata refreshes. One expensive or broken resource should not cause a global queue to stop progressing.

There is also an oscillation problem. Controller A observes a desired replica count of eight and increases an external pool. Controller B sees high costs and reduces it to six. If both believe they own the same field, they may fight forever while each reports successful reconciliation. Establish clear field ownership, policy priority or a single authoritative desired-state source. When independent controllers must cooperate, define compatibility rules and observable state transitions. Backoff alone cannot resolve contradictory goals, it merely slows the argument.

Leader election helps if only one controller replica should act on a particular resource. But leader election by itself does not prevent duplicate external effects: an old leader may pause, lose the lease, then resume and finish an operation already started. Use fencing, operation identifiers or provider-side conditional writes when side effects require exclusive ownership. Many Kubernetes controllers are safe with concurrent replicas only because of idempotent reconciliation and API concurrency controls. Determine which consistency contract the external resource actually provides.

Consider fairness and ordering. If one invalid resource causes a worker to retry continuously, it must not starve 10,000 healthy resources. Rate-limit retries per key, use a dead-letter or human-attention state where appropriate, and prioritize urgent safety reconciliation without creating an unbounded priority class. If all controllers resynchronize every ten minutes on the same clock boundary, randomize resync timing. A central workqueue can provide shared backpressure, but a single global queue across unrelated resource types can introduce its own failure domain. The right partitioning follows cost and operational ownership.

A good recovery exercise shuts down the API server, lets desired state change while watches are disconnected, and brings the API back under reduced capacity. Measure list burst rates, queue age, reconcile success per minute, retry attempts, resource-version relists, and actual convergence time. Inject an external provider call that succeeds while the response is lost. Verify no duplicate resource appears. Then introduce conflicting controllers and make sure the system detects ownership drift instead of thrashing silently. The interview answer should show why controllers are deliberately repetitive yet must not be wasteful: reconciliation gives eventual correction, while batching, ownership and bounded work prevent the repair mechanism from becoming the outage.