Every customer has three workers. Why do the same noisy tenants keep colliding?

Interview question. A job service operates 30 interchangeable workers and supports thousands of tenants. A tenant's jobs can overload workers. Static partitions leave some workers idle and ordinary random routing spreads one bad tenant's traffic widely. Explain shuffle sharding, choose the shard size and calculate what isolation it does, and does not, provide.

Imagine a support call from tenant North. Their exports have been slow for an hour. The on-call engineer sees no problem with North's requests. Another tenant has started a large scraping job that consumes CPU on their shared workers.

A straightforward first attempt is static partitioning. Divide the workers into ten groups of three and assign each tenant to one group. That bounds an overloaded tenant to three machines, but every tenant in that group is affected. Tenants may also distribute unevenly by actual work, leaving idle groups next to hot ones. A completely random worker selection on every request has the opposite problem: it uses capacity flexibly but permits noisy work to touch the whole fleet. Shuffle sharding looks for a middle ground by giving each tenant a stable, relatively small, distinct subset chosen from a larger worker pool.

Think of dealing cards. From 30 workers, assign each tenant a set of three distinct workers using a deterministic keyed placement function. For example tenant A receives workers 2, 11 and 28, while tenant B receives workers 4, 11 and 20. They share one worker but not their entire allocation. Keep the chosen set stable across requests so a tenant does not wander through all 30 machines. A routing library can derive the same assignment from tenant identity and a versioned membership map.

Here is the calculation that matters. With 30 workers and three in a hand, there are C(30,3) = 4,060 distinct hands. If two tenants were assigned uniformly and independently, the probability they receive exactly the same three workers is 1 / 4,060, around 0.0246%. That's encouraging, but it is not the probability they share any worker. The chance of at least one common worker is 1 - C(27,3)/C(30,3), about 27.96%. A noisy tenant saturating all three of its workers can affect another tenant on a shared worker, even when their hands differ. The size and severity of that effect depend on admission budgets and how the second tenant's work is scheduled over its healthy choices.

Do not tell the interviewer shuffle sharding makes collisions impossible. It reduces correlated blast radius when the allocation and scheduling policies make full dependence on all shared resources unlikely. If each tenant needs only one healthy worker for a task and can route around an overloaded choice, a partial overlap is less damaging. If a job requires all three assigned workers to coordinate, a single common hot worker might be enough to stall progress. The algorithm's isolation properties cannot be separated from the workload's execution semantics.

How should work choose among the three workers? For stateless requests, a client or router may pick the healthiest member of the tenant's hand with a bounded policy and avoid sending extra requests into a known full queue. For stateful workers, jobs may have sticky ownership or cached local state. Randomizing every task inside the hand may balance arrivals but can still create unfairness without per-tenant admission. Give each tenant a budget and make workers enforce local capacity limits. Shuffle sharding is not a substitute for rate limiting, worker fairness, authentication, or task cancellation.

What if ten large tenants share a small hand size but the rest are tiny? A uniform three-worker allocation may be a bad fit. You can assign differentiated hand sizes for customer classes, or move heavy tenants to dedicated capacity, provided you can quantify how those changes affect overlap and total utilization. Larger hands allow a tenant to use more worker capacity and tolerate individual worker failures, but also increase its potential interference footprint. Smaller hands improve isolation yet cap the tenant's peak parallelism. When fleet membership changes, deterministic placement may remap many tenants unless the assignment scheme is designed for stable membership updates.

Now imagine a worker dies. Every tenant whose hand includes that worker suddenly has only two available choices. Their tasks may shift simultaneously onto the remaining members. Health-aware routing and spare capacity need to handle this correlated movement. Replacing a failed worker can temporarily change a hand, but the policy must not accidentally give a hot tenant access to the entire fleet during recovery. Test how the system behaves when one worker fails, when all of a single tenant's workers are saturated, and when an entire availability zone is unavailable. Worker placement across zones matters more than neat combinations on paper.

It's worth contrasting shuffle sharding with a cell. A self-contained cell has separate data-plane resources and, ideally, independent state. Shuffle sharding is usually a subset-allocation strategy inside a shared pool. It allows controlled overlap and better packing but therefore does not promise the hard failure boundary of a cell. You might place several cells across a region and use shuffle sharding within each cell for stateless workers. Do not shuffle workers across cells if doing so destroys the independence you built the cells to provide.

For an interview, I would show the collision arithmetic, then ask how much useful work a customer loses when one shared worker gets hot. Measure the affected-tenant count, worst-case overlap, per-tenant p99, worker CPU, queue age and fairness under deliberate noisy-neighbor load. A product might accept a 28% chance of touching at least one common worker because the other two still provide healthy capacity. Another product might not. Shuffle sharding becomes an engineering choice only when you connect the combinations to actual customer isolation and capacity contracts.