Interview question. A delivery analytics topic has 24 Kafka partitions and one consumer group. Processing is slow, so the team increases consumer instances from 24 to 80. Total throughput barely changes. One partition holds most of the lag, and almost all records on that partition use the same large carrier ID. Explain what additional consumers can and cannot fix. Propose options that preserve the ordering promise if the carrier insists on it.
Assume each active consumer instance processes records from its assigned partitions using a normal serial application path. We can change the data model and consumer implementation, but cannot silently discard messages or pretend two dependent events are independent.
Think first
A topic has 24 partitions. You scale one group from 24 to 80 consumers. One partition gets 18,000 messages per second and one worker completes only 4,000 per second. Diagnose the backlog and propose changes if ordering is required per shipment but the partition key is carrier ID.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
Count work owners, not pods
Kafka assigns each partition to at most one member of a conventional consumer group at a time. With 24 partitions, increasing the group to 80 ordinary consumers does not make 80 independent partition readers. Many members have no partitions. More servers can help only if the existing work can actually be assigned or processed in more parallel units.
This is not primarily a CPU-autoscaling question. First inspect assignments, per-partition lag, incoming and processed rates, consumer poll and processing time, group rebalances, downstream latency and the distribution of keys. An overall lag number hides the one partition that is dominating the incident.
Twenty-four partitions are assigned to at most twenty-four group members while one hot partition dominates the backlog
Suppose the hot partition receives 18,000 records per second and its effective processing capacity is 4,000 per second. Its backlog grows by 14,000 per second while conditions remain constant. In a minute that is 840,000 more records. Adding 56 idle consumers changes neither side of this equation.
A more accurate diagnosis also asks whether the worker is CPU bound or spending most time in a database, schema decoder, external API, lock, or output queue. If one worker can safely process more records per batch, improve that before redesigning the partition key. But no tuning can beat the actual serial dependency rate without changing how the work is organized.
One carrier is not necessarily one unit of ordering
The topic was originally keyed by carrierId. The team assumed all events from a carrier must be ordered. But the product question may only require ordering for each shipment handled by that carrier. Those are very different contracts.
If event A updates parcel 11 and event B updates parcel 97, nothing in the problem says B must wait for A. Publishing by shipmentId or a composite of tenant and shipment could distribute the work across partitions while preserving per-shipment order. That changes the mapping and may require the controlled migration from the preceding lesson. A direct partition increase alone will still map the same carrier ID to one partition under a stable key function.
Do not declare victory because keys are evenly distributed by count. One exceptionally active shipment could still dominate its key. Examine the work rate and its cost per key, not merely the number of distinct keys per partition.
What if the carrier truly needs total order?
Ask what rule makes all its events dependent. Perhaps a regulatory ledger requires a single serial sequence for that carrier. If each update must examine and advance a carrier-wide invariant, sharding its messages across consumers without a new coordination protocol would weaken the guarantee.
There are choices, each with a cost. Optimize the single owner through bulk state transitions and faster storage. Coalesce operations only when the business semantics permit it. Split the data into independent sub-ledgers with a later ordered reconciliation point, if the invariant can be defined that way. Or accept a throughput ceiling and apply admission upstream rather than letting lag grow indefinitely.
If the carrier needs a presentable order but operations commute, the system might process subsets in parallel and publish a sequenced summary afterward. That is not the same as preserving an order-dependent write path. Interviewers will want to know which guarantee remains true after the optimization.
There is also an implementation option for per-key concurrency inside a partition: read a batch, dispatch work to distinct per-key queues, and track a safe completed prefix for broker progress. This can use spare CPU even when the partition count is fixed. But the processing receipt and output rules must protect against redelivery. Saving the largest completed offset while a smaller record is still running can skip that smaller record after a crash. Here the engineering decision is identifying genuinely independent work while keeping broker progress recoverable after a crash.
Measure the real result
A successful change should reduce the hot partition's oldest-event age and improve useful completed events per second, not merely lower the average consumer CPU. Track lag by partition, throughput by key range, error and rebalance rates, p95/p99 processing age, and downstream saturation. During a cutover from carrier-keyed to shipment-keyed events, monitor ordering violations and missing sequence gaps rather than trusting total throughput alone.
Suppose the new scheme processes 22,000 records per second while the carrier publishes 18,000. That creates 4,000 per second of spare capacity on the relevant workload, before accounting for other traffic. If the backlog already reached 14.4 million records, clearing it takes an hour under the simplified constant-rate model. The platform should communicate that recovery time instead of interpreting a non-growing lag as a recovered service.
The answer is not "add partitions" or "add threads." It is to discover the actual unit of independent work, protect its ordering contract, and then use that freedom to parallelize. When there is no such freedom, capacity is a business constraint to expose.
Now suppose the partition assignment is healthy, but one particular event can never be decoded. Do we let it stop every event behind it?
Go deeper
The 24 partition limit describes conventional group assignment. Custom parallel processors can run multiple independent tasks after polling a partition, but cannot bypass the required ordering and safe offset progression. In Kafka, consumer pause and resume can manage fetch flow without changing group membership, provided polling and rebalance behavior remain correct. If one carrier monopolizes a single physical partition, adding producers or consumers is not a substitute for changing the partitioning contract. Watch for consumer group changes and reassignment churn while measuring improvements.
Check your reasoning
Explain why conventional group parallelism is bounded by assigned partitions and why extra members can be idle.
Calculate the growing backlog at 14,000/s and identify per-partition metrics and downstream bottlenecks.
Clarify whether the ordering invariant is per carrier or per shipment before resharding.
Describe safe per-key parallelism or better partitioning and why the same hot key remains one routing unit.
Distinguish total throughput from backlog drain time and truthful recovery status.
Common mistake: Assuming any number of consumer pods increases parallelism, or claiming one globally ordered key can be arbitrarily split without coordination.
Read the answer explanation
14,000 messages per second. — Incoming minus completed = 14,000/s of additional backlog on that partition.
Apply it elsewhere
One tenant owns 70% of a vector-index ingestion topic, but updates only need order by document ID. What key would you consider?
Compare your approach
Repartition by document identity or independent logical bucket while preserving per-document sequencing.
One more constraint
After fixing key skew, throughput exceeds arrival rate by 4,000/s and backlog is 14.4 million. How long to clear under steady rates?
Quick revision
Conventional consumer-group parallelism is bounded by partitions. With 24 partitions, 80 members do not create 80 readers, and a hot key can dominate one assigned partition. Diagnose key skew and the actual worker bottleneck. The hot lane grows by 18,000 minus 4,000, or 14,000 records per second. Scale by the genuine unit of independent work while preserving its ordering contract. Otherwise expose a hard capacity limit.
