Interview question. A delivery platform publishes parcel scans into Kafka. The producer team increased batching to reduce network requests and broker CPU. Total throughput improved, but customers now see an occasional 180 ms delay before a time-sensitive ParcelArrived event even reaches the broker. Which parts of the producer would you inspect, and how would you choose a batching policy without giving up its throughput benefit?
Assume the producer is asynchronous, the broker is healthy unless evidence says otherwise, and the customer-facing freshness objective is measured from the business event to the downstream projection, not just from send() to a callback.
Think first
Parcel scan publication became cheaper after raising linger to 200 ms. Quiet depots report stale customer-facing status. Show what you would measure to isolate the delay and propose a batching policy for routine telemetry versus urgent dispatch events.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
A fast API call is not a published event
An application calls producer.send(record) and it returns quickly. That often means the client accepted the record into a local buffer. It does not mean the broker has stored it, or that the consumer has read it. The callback or future reports a later result under the producer's configured acknowledgment contract.
The team reports a benchmark: "Our produce API now takes only 0.2 ms." That benchmark might only time enqueueing. I would first instrument four timestamps, or equivalent spans, with a shared event ID: event creation, accepted into the producer, broker acknowledgment, and consumed for the customer-facing view. Otherwise a fast enqueue can hide a growing local queue.
In Kafka, producers maintain batches by partition. A record may wait for a larger batch. The linger.ms setting places an upper bound on the intentional batching wait in normal conditions, and the batch can leave early when it fills. Broker backpressure, request delays and client congestion can extend the observed wait beyond that setting. The Kafka producer configuration documents both the buffering behavior and its limits.
A quiet partition holds a record for the linger window while a busy partition fills and sends early
Look at quiet partitions before arguing about throughput
Imagine a producer configured with a 200 ms linger. At a quiet site, only one parcel event is published every few seconds. Its batch rarely fills, so the record may incur much of that optional waiting time. At a busy sorting facility, dozens of events for the same partition can fill a batch quickly. The same setting produces very different latency distributions.
If the observed extra delay is 180 ms, that does not prove linger.ms is the cause. Compare buffer wait, request queue time, broker acknowledgment latency and producer throttle time. Also inspect retries and compression CPU, which can differ between the quiet and busy facilities.
Use traffic with realistic key distributions, not only one hot partition that constantly fills its batches. Measure batch-size distribution, records per request, bytes transferred, compression ratio, broker utilization, end-to-end p50 and p99, and deadline misses. An average record latency improvement can still conceal a bad experience for a quiet customer.
Choose the policy from the work, not from a benchmark trophy
The producer is carrying two kinds of data. Routine location telemetry can tolerate a little delay because the customer interface aggregates it. A dispatch decision might have a strict deadline because another service must allocate a driver immediately.
I would establish separate freshness contracts for those two event types. Use separate configurations or streams where needed. The urgent path can prefer smaller linger, predictable bounded buffering and an explicit admission policy. The bulk path can choose more efficient batching and compression when its freshness contract permits it.
Why not force linger.ms=0 everywhere? Because batching can reduce per-record network and broker overhead. At high throughput the producer may naturally batch even with zero intentional linger, but at low throughput it may send small requests. We should pay that cost only where the latency objective justifies it.
batch.size is not a minimum amount that must be collected before any record can leave. It is a batching-size control. A larger allocation can increase memory consumption, particularly when many partitions are active. buffer.memory and max.block.ms then matter when the broker stops accepting data as quickly as the application produces it. Blocking forever at the producer boundary is not a harmless response to overload.
Decide what a deadline means
Suppose an urgent event has a 300 ms business freshness target. A client-side linger of 200 ms might use most of the budget before network, broker, fetch, application processing and projection update take place. It would be poor design to meet a 300 ms broker-ack metric while consistently missing the customer-visible deadline.
Build an explicit budget for the entire path. If the broker acknowledgment itself is mandatory before the caller proceeds, choose the API behavior accordingly. If the caller is allowed to move on while publication is pending, preserve the record and make the pending state observable. Those are product and durability choices, not just configuration changes.
When the producer is overloaded, a lower linger does not create additional network bandwidth or broker capacity. It can result in more requests and less efficient compression. A bounded queue and a visible failure or slowdown are preferable to unbounded memory use. For events whose business effect cannot be dropped, the admission path must either retain them durably through an agreed mechanism or reject new work before promising acceptance.
I would investigate before turning the knob. Trace buffering and acknowledgment, distinguish quiet from busy partitions, define end-to-end freshness, and tune batching against evidence.
Try this change: The producer is now fast, but downstream consumers still update the dashboard 500 ms late. Would shrinking linger again fix it? Not unless measurements show producer waiting remains the limiting part. We have fixed one stage, not the pipeline.
Next, the team tries to raise throughput by doubling the topic's partitions. Read why repartitioning can reverse per-entity updates for that failure boundary, then continue with the next question on consumer parallelism.
Go deeper
Producer batching depends on the number of active partitions, key distribution, record size, compression and in-flight requests. Compare record-queue-time-avg, batch-size-avg, request-latency-avg, throttle metrics and end-to-end consumer visibility for the client and broker versions you run. Sampled timestamps and broker offsets can reveal the difference between producer buffering and downstream lag. delivery.timeout.ms includes batching, request acknowledgment and retriable failures. It is not a customer-visible latency SLO, and client retries after ambiguous outcomes still need an operation contract appropriate to the business effect.
Avoid building a benchmark that produces continuously to one partition, then assuming its batch fill times apply to a quiet customer. Make a small controlled matrix of event sizes, active partition counts and traffic patterns. Document the chosen separation between urgent events and bulk telemetry before promoting a configuration change.
Check your reasoning
Separate application enqueue time, producer queue time, broker acknowledgment and customer-facing visibility.
Explain that linger is a bounded intentional wait under normal conditions and active partitions can fill early.
Compare quiet and bursty workloads with realistic keys, record sizes, batch fill and compression measurements.
Propose distinct latency and throughput policies for urgent dispatch and routine telemetry with bounded producer memory.
Explain what the producer can honestly promise when backpressure prevents timely publication.
Common mistake: Believing send() returning means the broker durably stored an event, or setting linger to zero as a complete overload solution.
Read the answer explanation
The producer may intentionally hold the record for much of the window before sending, with other delays still possible. — The linger wait is an opportunity for batching, not a commitment that downstream processing finishes at the deadline.
Apply it elsewhere
A fleet of AI trace exporters starts batching logs. Debug traces now arrive late during incidents. Which policy boundary would you change?
Compare your approach
Separate observability freshness from bulk throughput, measure actual buffer and collector delay, and avoid unbounded local memory.
One more constraint
If producer acknowledgment time is 15 ms but consumers update the view after 700 ms, where do you investigate next?
Quick revision
A fast send() usually establishes local enqueue, not broker storage or consumer visibility. Kafka batches per partition, and linger.ms may delay quiet records while busy partitions fill early. Measure the whole path, vary realistic workloads, and choose separate freshness and batching policies for urgent dispatch versus bulk telemetry. A strict end-to-end deadline must include downstream processing. Bounded buffers and admission matter when the broker falls behind.
