Skip to content

The writes are spread evenly. Why does one customer's history need 80 queries?

The shard that moved · Part 3 of 5 · 6 min read

Read the worked explanation ↓

The interview question. You are designing an event store with 80 physical shards for thousands of business customers. It must support a point lookup by event ID and a time-ordered scan of all events for one customer. Customers range from a few events daily to millions hourly. Compare hash, range, and composite partitioning, including a very large customer.

Assume events are immutable once accepted, the customer ID is known for time-range queries, and a globally ordered history across different customers is not required. The database may support an index or lookup table to find an event by its ID. Do not assume that any one vendor automatically indexes both access paths.

Think first

Build a multi-tenant event store with point lookup by event ID and last 30 days of time-ordered events for any customer. One tenant generates 40 percent of writes. Compare hash(event ID), tenant-prefixed range layout, daily buckets, and per-day sub-buckets. Show the read cost you accept for each option.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

An even hash does not preserve the query you care about

You can hash each event_id to spread inserts evenly. It is simple and works well for direct reads when you know the ID. But consider the request “Give me customer T42's events between 09:00 and 10:00, sorted by time.” Events from T42 are scattered across many partitions. The query may fan out to all 80 shards and merge results. Its latency depends on the slowest participants, and each new partition adds coordination work.

At the other extreme, you can store every customer's events together, ordered by (customer_id, event_time). The customer's range scan becomes efficient. The catch is a whale tenant. If one customer produces 40 percent of writes, its contiguous range may be served by one physical unit of capacity even though hundreds of smaller customers are spread nicely.

Hashing individual event IDs spreads writes but scatters customer history, while tenant-prefixed ordered keys allow local scans until a large tenant needs time buckets and sub-buckets
Hashing individual event IDs spreads writes but scatters customer history, while tenant-prefixed ordered keys allow local scans until a large tenant needs time buckets and sub-buckets

Neither layout is universally correct. Measure the mix of access patterns, scan sizes, and whether point lookups can tolerate an index hop.

Use a customer prefix, then order within it

A common starting point is to hash the customer identifier for broad placement, then maintain records in a local ordered structure under that customer. Logically the key is (tenant_id, timestamp, event_id). The tenant portion routes the operation, while the ordered suffix supports range scans without looking at unrelated tenants.

The stable event ID breaks ties between events with identical timestamps and supports repeatable pagination. If the caller needs the same results after new events arrive, its pagination contract must define whether it reads from a fixed snapshot or advances through a changing stream. Sorting by timestamp alone is not enough for a total order when clocks disagree or events share timestamps.

DynamoDB explicitly distinguishes a partition key from a sort key and uses the sort key to support range queries within a partition-key value. Bigtable instead emphasizes designing lexicographically ordered row keys around prefix and range scans. Those are related design principles, not identical implementations. DynamoDB sort-key guidance and Bigtable schema design.

The ID lookup now needs attention. Either the client already knows the tenant, the event ID encodes enough routing information, or you maintain a separate index from event ID to the owning tenant and location. That extra index has its own consistency and recovery behavior. An answer that delivers perfect time scans by silently abandoning point lookups has not solved the original problem.

A large customer may need several physical ownership ranges

For moderate tenants, one tenant-level group may be a good fit. For a customer accounting for 40 percent of writes, consider bounded time buckets, such as (tenant_id, day) as a routing unit. Historical days become relatively cold, and independent days can be placed separately. A week-long request becomes seven range queries and a merge, which is often an acceptable price for distributing the data and simplifying retention.

But daily bucketing does not automatically spread today's live writes. All of today's inserts may still target the current bucket. If that bucket exceeds a partition's capacity, split it further with a sub-bucket derived from a stable property, or by explicit range ownership. A query for that day must then query the relevant sub-buckets and merge their ordered results.

For example, four sub-buckets could each receive one quarter of this customer's writes under a good assignment. That is a capacity hypothesis to verify, not a guarantee. If events must be processed in one strict acceptance order, independently accepting writes into four buckets introduces another question: who assigns the order and how does a reader recover it? A timestamp sort is only chronological ordering according to the timestamp's semantics, not proof of one globally serialized acceptance sequence.

Bigtable warns against a timestamp-first row key because monotonically increasing prefixes can concentrate writes on one tablet. Appending a timestamp after a well-distributed prefix is better for broad distribution, but even that can leave a single extremely active prefix hot. Bigtable schema best practices.

Leave yourself a way to change placement

Do not bake the partition count into permanent event IDs if a later reshard would force clients to reinterpret them. Treat physical placement as versioned metadata that can evolve while event identity stays stable. Document whether a customer is routed as one unit, by day, or by day and sub-bucket.

Measure per-tenant write share, hottest physical partition, query fanout, merge cost, and the number of active ranges per tenant. Once a tenant's layout changes, range readers must understand both historical and new layouts until backfill or a stable directory makes them uniform.

You have designed a partition key that supports useful reads. What happens when the one tenant that was small last year now needs to move to a different database while writes continue? The data layout is only half the migration. The other half is who is allowed to accept the next write.

Go deeper

Go deeper: locality, sorting, and operational semantics

A composite logical key and a physical shard key are different things. (tenant_id, event_time) in a B-tree does not guarantee that all rows with the same prefix fit on one physical node in every distributed SQL database. Some systems automatically split ranges. Others hash only the partition-key attributes. The exact operator semantics need to be named when you choose a product.

Range scans with time buckets require stable pagination across buckets. A cursor may need the last (timestamp, event_id) and the layout version, or a snapshot identifier, rather than only an offset into one node's results.

Sub-bucketing by hash(event_id) gives more write spread but makes a per-customer ordered scan a scatter-and-merge task within the requested period. Bucket by a domain dimension only if that dimension really supports the application's query patterns. Otherwise the supposedly convenient assignment can become a permanent access-path tax.

Retention also matters. Dropping expired whole-day partitions can be much cheaper than deleting individual old rows, but only if time buckets are compatible with late-arriving events and legal hold requirements. A late event should be routed according to its documented event-time or ingest-time rule, not whichever bucket happens to be open.

Check your reasoning

  • Separate routing by event ID from routing by tenant and ordered suffix.
  • Use composite tenant and timestamp ordering to support targeted range scans under the stated database semantics.
  • Preserve point lookup through known tenant identity or a separately maintained event-ID index.
  • Explain why a time bucket does not automatically distribute the current hot bucket and when sub-bucketing helps.
  • Name the fanout, merge, strict ordering, and online key-evolution consequences.

Common mistake: Claiming hashing and time ordering are simultaneously free, or saying that one daily bucket solves a hot tenant writing only into today.

One tenant causes 40 percent of writes and all new events go into its current daily bucket. Does splitting historical days solve the live hotspot?

Read the answer explanation

No. The live bucket may still be a single hot ownership range. — Time bucketing separates periods, not concurrent work inside the current period.

Apply it elsewhere

An audit API needs every change to one repository in commit order, but a huge repository dominates the write path. What must you clarify before dividing its log?

Compare your approach

Ask whether order is required for each file, each branch, or the entire repository. Different ordering scopes permit different partitioning strategies.

One more constraint

The tenant wants a single strict acceptance sequence after sub-bucketing. What new authority or sequencing cost appears?

Quick revision

Hashing individual event IDs spreads writes but scatters all events for one customer. A tenant-prefixed composite key keeps that customer's time ranges together while supporting local ordered scans. The largest tenant may exceed one partition's capacity, so time buckets and, if necessary, sub-buckets can divide the ownership range. Those changes increase read fanout and may require a separate event-ID index. Strict acceptance order cannot be invented by merging unsynchronized timestamps. Treat placement as versioned metadata so it can evolve.

References