One tenant published three times more. Why did every team slow down?

A healthy broker can still be unfair

Shared brokers compete for several resources: network bandwidth, request handling threads, disk throughput, log replication capacity, page cache, and sometimes partition-leader concentration. A producer that publishes three times as many bytes may saturate a network or disk path shared with other clients. Even when global CPU is moderate, a single busy broker or disk can be limiting.

I would not start by increasing every broker's thread count. I would identify the constrained resource and which tenant, topic, partition and broker account for the change. Compare producer throughput, produce/fetch request latency, request throttle time, replication lag, disk and network saturation, consumer oldest-event age, and request queues.

Kafka supports client quotas covering produce and fetch byte rates and request capacity, with identity and client-based enforcement in its multi-tenancy documentation. Those are useful controls, but broker-side quotas are not an entire end-to-end fairness policy. Consumers on a shared hot partition can still experience contention.

Define the minimum service each customer deserves

Suppose brokers can sustainably handle 120 MB/s of this workload while preserving necessary replication and operational headroom. Normal traffic is 90 MB/s, including 35 MB/s from this tenant. Tripling its rate raises total traffic to 160 MB/s. Merely adding producer retries or allowing application buffers to grow does not create the missing 40 MB/s. We need to reduce admitted load, redirect it to verified spare capacity, or add capacity through an explicit plan.

For a shared tier, I would set authenticated tenant budgets for steady throughput and short bursts. Use producer identities that cannot simply invent another client.id to escape the contract. Reserve capacity for platform maintenance and for critical streams where the product requires it. Classify what may be slowed or rejected instead of declaring that every record has absolute priority.

A high-priority tenant may deserve a larger reserved or paid capacity tier. That decision belongs to a published service contract. Staff and Principal reasoning includes what happens to other teams, how exceptions expire, and who reviews the total capacity decision.

Backpressure must reach the real source

If the broker throttles a producer, the application may continue enqueuing events locally. Once its buffers fill, send() may block or fail under producer configuration. A service that already told the caller "accepted" must have an appropriate durable handoff for work it cannot immediately publish. Otherwise the backlog is merely hidden in a process that can crash.

For discretionary telemetry, the service might sample, combine, or drop observations under an explicit loss budget. For an accepted business operation, that is usually not valid. Its upstream request should receive a bounded wait, a rejection before acceptance, or a durable work-acceptance path with a truthful status and recovery contract.

I would place admission where source volume is known: tenant-aware gateways or publisher SDKs can enforce quotas before serializing and buffering expensive records, while brokers act as another protection boundary. Apply bounded memory and request rates at every stage. Do not use a single unbounded shared queue, where one tenant can consume all queue slots and make unrelated operations wait.

Some workloads require physical isolation

Software quotas cannot eliminate every interference path. A single hot topic can concentrate partition leaders on a broker. Large messages can create spiky disk and network costs. Recovery replication may contend with foreground production even when produce quotas are functioning correctly.

For especially important workloads, consider separate topics, broker placement, storage policies, or isolated clusters where the observed contention and business guarantees justify the additional operating cost. If two teams have incompatible retention, latency or access-control requirements, separation may be simpler than an increasingly complex set of exceptions.

The decision should follow measured resource pressure. Moving an entire company to one cluster per tenant without evidence creates large metadata and operational costs, and it may solve the wrong problem.

Design a test where everyone is watching

Reproduce a single-tenant 3x burst while measuring the target tenant and two unrelated tenants independently. Prove that admitted work stays within the promised shared capacity, that critical traffic preserves its objectives, and that throttled clients do not cause new resource exhaustion in their applications.

Also inject a broker failure during the burst. Capacity that works only when every replica and disk is healthy is not necessarily the platform's usable capacity. Quotas should be reviewed against degraded capacity, with a clear priority policy for when the total available throughput drops.

Enabling Kafka quotas is one step. Map who consumes each limiting resource, enforce fair admission at several boundaries, and decide which work may wait or be rejected. Shared infrastructure needs an explicit contract for whose work wins under contention.

Transfer question: A shared GPU inference cluster has one tenant sending long prompts while others submit short interactive requests. What changes? Replace byte-rate as the main cost signal with compute, KV memory and latency budgets, while retaining the same fairness and admission reasoning.

This closes the six-lesson sequence. The questions moved from buffering, through routing and processing, to historical retention and shared capacity. The next useful skill is being able to ask which guarantee each stage actually provides, before combining them into one end-to-end system.