One tenant's import crashed a shard. Why did half the company go offline?

Interview question. A B2B analytics platform hosts 40,000 customers in one region. Every customer has a tenant ID, but all requests eventually reach the same databases, workers and metadata service. A large import by one tenant destabilizes the platform. Design cell-based isolation and explain what has to be duplicated, shared or routed for the isolation to be real.

The team thought it already had tenancy isolation. Every database row carried tenant_id, authorizations checked the tenant, and each customer's tasks used a separate logical queue. During a large customer's overnight import, the ingestion database ran out of write capacity. Search requests for other customers slowed down. A few worker processes ran out of memory. Then a rollout of emergency throttling rules hit everyone at once.

A cell changes the unit of failure. Instead of one giant shared execution and storage environment, build multiple copies of the workload stack. A customer is assigned to a cell, and the cell contains enough request serving, background execution and state management to operate without borrowing critical runtime resources from another cell. A failed import in cell B should not reduce cell A's ability to process customer requests. That claim is stronger than “we put tenant B on another database partition.” It includes the supporting services on the critical path.

Begin by finding the existing blast-radius graph. The supposedly independent queues all feed one worker cluster. The databases are partitioned, but share a single storage I/O budget. The authentication service verifies local credentials but performs a synchronous centralized entitlement lookup. Every cell would still depend on those resources if we copied only the API servers. A cell must not contain cross-cell runtime calls, and its data plane should usually be able to serve known customers with a temporarily unavailable configuration control plane.

How big should a cell be? Suppose the entire platform handles 80,000 request-cost units a second. Benchmarks show a cell can sustain 12,000 units a second at the desired p99 under fault tests. Eight cells offer 96,000 nominal units, but 16,000 spare units across a region are not automatically an N+1 guarantee. A customer placed on one overloaded cell does not benefit just because spare capacity exists elsewhere. Nor can a region survive losing one whole cell unless routing, state placement and admission policy permit affected customers to move safely, which may be neither quick nor desirable. Pick the cell limit from observed resource saturation, then enforce onboarding and per-cell operating headroom.

Routing is a subtle dependency. A stateless edge receives tenant-42 and needs its assigned cell. If it consults one metadata database for every request, losing that database makes all cells unreachable even though the cell data planes are healthy. Instead, distribute versioned tenant-to-cell maps and let routers operate on a bounded-staleness local snapshot. An unknown tenant or a security-sensitive reassignment cannot be guessed. Decide what happens when a cached mapping expires, and prevent a stale router from delivering writes to an old owner during migration. The fencing and ownership principles from earlier partition-migration lessons still apply, but the new architectural question is keeping the routing plane from coupling independent cells.

A second trap is assuming shared credentials, deployments and storage accounts are harmless. A region-wide bad deploy can corrupt every cell if all cells receive it simultaneously. Shared KMS limits or one secret store can block each cell's startup. A single multi-tenant bucket may be sufficient if it offers independent performance and protection for the intended failure class, but it still creates a shared resource and security domain that must be made explicit. Isolation is always relative to the failure you're trying to contain. You cannot promise that every provider-wide outage will stop at a cell boundary.

Cells also complicate customer migration. Moving a tenant isn't just changing an edge map. Stateful data, scheduled work, queued tasks, deduplication state, audit history and long-lived streams may live in the old cell. Define a migration protocol that snapshots state, catches up changes, blocks or redirects old writes, changes ownership with an epoch, and verifies read and write behavior before cleanup. Large migrations should be throttled so that moving a few premium accounts does not exhaust both source and destination. Some customers may require an approved maintenance window rather than a seamless live cutover.

What happens to platform-wide functions? Global search across all customers may need explicit fanout with a bounded budget or, more often, a separate aggregate view. A global administration portal can use an asynchronous inventory of cell metadata. Billing reconciliation can consume per-cell events with provenance rather than asking every cell for a synchronous response during checkout. These global features should not casually reintroduce a mandatory runtime dependency among the cells. Similarly, limit emergency rollout propagation. A problematic config should be applied to one canary cell, observed, then deployed in staggered waves with an independent rollback path.

The business tradeoff is real. Cells duplicate fixed overhead, reduce average resource pooling efficiency, and require placement, migration, per-cell SLOs and operations tooling. For a small service that rarely saturates and has few tenants, basic per-tenant quotas may be cheaper and sufficient. For a large SaaS where one customer's behavior should never threaten another, that overhead purchases a valuable failure boundary.

For acceptance, run one tenant's adversarial import until its cell rejects excess work or becomes unhealthy. Verify that other cells' p99, error rates, data correctness and deploy capacity remain within independent targets. Then disconnect the global map control plane and check that known tenants still work. Finally, deploy a deliberately bad change to one canary cell and verify that the rollout stops before crossing the intended boundary. If those tests fail, the boxes on the diagram are not yet cells in the operational sense.