The routing config passed every check. Why did one cell receive 70% of traffic?

Interview question. A company operates eight independent application cells. A new tenant-routing configuration passes schema validation and deploys without errors, but within five minutes one small cell receives 70% of request traffic and begins rejecting work. Design validation and staged rollout so syntactically valid routing changes cannot silently violate placement and capacity safety.

A route table maps tenants to cells using identifiers, destinations and a revision. Someone adds a default route for unknown tenants and merges the change. All records parse. Destination cell names exist. The control plane distributes the bundle successfully. Then one cell becomes saturated, the others are mostly idle, and the platform team discovers that the new fallback mapped an entire fast-growing customer cohort to the same small deployment. This is a semantic failure.

The natural first fix is to add a test that rejects any config directing more than 20% of tenants to one cell. But tenant count is not traffic. One large tenant might generate more work than 2,000 small tenants. Some requests are read-heavy, others consume database writes or expensive GPU work. A safe routing validator needs a model of load units, failure domains, and hard constraints, not just a parser for a mapping file.

Start with the invariants that cannot be approximated: tenant residency, access boundary, a tenant's unique authoritative owner for writes, and any isolation requirements attached to regulated customers. Validate that every known tenant maps to exactly one permitted cell for a given routing epoch. If the architecture intentionally supports replicas or active-active operations, that needs a separate explicit contract. A generic fallback must not silently assign a tenant to an unauthorized region, even when the destination has plenty of capacity.

Then estimate load from recent measured behavior. Suppose the cell can sustain 12,000 normalized work units per second at the chosen latency target, and a promotion would add 9,000 to an existing baseline of 5,000. The new total is 14,000, exceeding the example capacity by 2,000, or approximately 16.7% of that capacity. Reject the candidate or require an explicit capacity change. Don't assume measurements are perfect: record confidence intervals, workload forecasts, burst envelopes and how missing tenants are represented. Treat unknown and worst-case tenants conservatively rather than optimizing solely for average utilization.

Next simulate the difference between active revision 86 and candidate 87. Which tenants move? Which cells become hotter? How many cross-region requests are introduced? Which caches turn cold? A move can be safe at the steady-state destination load yet harmful during migration because both cells temporarily retain state and copy data. If routing authoritatively changes write ownership, the cutover requires a correctness mechanism: advance the ownership epoch and ensure the old owner cannot accept writes after the new owner becomes authoritative. A config update alone does not transfer an authoritative database safely.

Dry runs should use real enough distributions to find hot cohorts and routing skew without exposing private customer records. Hash or pseudonymize tenant identifiers, bucket workload by region and operation class, and replay sampled routing decisions without sending production writes to a candidate owner. Capture the resulting predicted utilization per cell, violations of tenant placement policy, and the set of changed assignments. Compare that to a control simulation that applies the currently active config over the same requests.

Now deploy the routing change gradually. Canarying is harder than a normal feature flag because an owner map often cannot split one tenant randomly across cells without violating state ownership. Promote selected tenants, or selected read-only shadow traffic if suitable, rather than assigning 1% of every tenant's writes to a second owner. If the migration involves a real ownership change, perform prepare, catch-up, fenced cutover and routing promotion as one coordinated protocol. A safe canary for router logic can also be a shadow evaluator that predicts choices while real requests use the old map.

Fleet consistency matters. At 14:00 some routers may hold revision 86 while others hold 87. If all endpoints are only routing read-only requests to equivalent replicas, brief mixed revisions may be tolerable. If routers direct authoritative writes to mutually exclusive owners, mixed revisions can be dangerous. Attach the ownership epoch to the request or otherwise ensure the storage layer rejects stale ownership. Never rely on immediate fanout delivery to 10,000 router processes as the sole correctness guarantee. Detect and alert on routers that fail to adopt a required revision.

Define objective stop conditions. For example, abort promotion if any destination exceeds a conservatively chosen sustained work-unit threshold, p99 diverges beyond the control cohort, queue delay increases, unexpected traffic reaches a forbidden region, or authoritative-owner conflicts appear. Measure effective traffic at the destination, not just routing decisions computed at the proxy. A router might believe it split traffic evenly while persistent gRPC channels still pin requests to one cell.

What about rollback? Returning routers to revision 86 may be safe if the only effect was a stateless traffic steering choice. It may not be safe if tenant data has already moved or a destination accepted newer writes. For stateful moves, rollback is a second coordinated migration with explicit ownership and reconciliation. Keep the last verified configuration in every router and an out-of-band emergency promotion path, but do not treat an old revision as automatically valid forever. A cell may have been retired, lost residency approval or exhausted capacity since that revision was created.

Finally run an incident exercise. Inject a routing rule that is syntactically valid but assigns a large region to one small cell. Verify that preflight simulation catches it. Then inject misleading metrics so the preflight passes and see whether staged production observations halt the push before broad impact. Also disconnect half the routers during rollback and verify that stale owners cannot accept forbidden writes. The key learning is that a config is program behavior expressed as data. Correctness requires semantic predictions, controlled exposure and enforcement at the resource that owns the invariant.