We quarantined the bad event. Which Kafka offset can we safely commit?

The first decision is not a retry duration

Before picking a backoff, determine whether the error is transient or deterministic. If the downstream database is temporarily unavailable, retries can be useful. If the payload has an unsupported schema ID, retrying the same bytes with the same code every second will not make them decodable.

An infinite retry loop turns one invalid record into a permanent availability decision for that partition. It can also waste broker fetches, worker CPU and operator attention. But skipping the event immediately can be equally dangerous if later records depend on it.

The question is: what may safely proceed while this event is unresolved?

Suppose parcel p42 has version 7 at the bad record, followed by version 8 containing a relative update such as "advance status from loaded to dispatched." Applying version 8 without knowing what version 7 changed could violate the state machine. Quarantining version 7 does not mean that version 8 has become safe.

Now consider parcel p91, whose events are independent of p42. The product may allow them to continue even while p42 is paused. That gives us a narrower failure boundary than blocking every parcel on the same Kafka partition. The system must still preserve correct progress and recovery behavior for both lanes.

Give an unrecoverable record a durable outcome

For a first implementation where correctness matters more than throughput, I would pause progress on the affected partition, store the failure evidence, alert an owner, and repair the decoder or publish an authorized correction. Processing resumes only when the missing transition is resolved. This is operationally expensive but easy to reason about.

If the business allows unaffected parcels to continue, build explicit per-parcel work lanes or another partition-independent state machine. Park p42 and every later dependent p42 event. Continue p91 only if its outcome is durably recorded and the consumer can recover it without skipping unresolved earlier work after a crash.

A dead-letter topic is useful for preserving the original record, schema information, failure reason, time, processing identity and redrive context. Confluent's Kafka Streams exception guidance discusses failure, skip and quarantine approaches. It does not make any of them universally correct for ordered business effects.

Crucially, publish to DLQ and commit the original offset are two separate actions unless a supported transactional boundary links them. For Kafka-to-Kafka processing, a suitable Kafka transaction can combine the output record and consumed offset under its guarantees. An external database, email provider, or human repair workflow needs its own durable protocol. Otherwise a crash between these actions may create duplicate quarantine entries or silently lose a record if progress is advanced prematurely.

Stable original event identity makes the quarantine and repair workflow easier to reconcile. A dead-letter topic is not an unattended graveyard that satisfies a delivery guarantee merely by receiving a record.

There are two distinct progress models. With completed-prefix tracking, commit only the next offset after a contiguous prefix whose outcomes are durable; replayed independent work must be idempotent. Alternatively, broker progress can acknowledge a durable handoff of every skipped or unfinished record into a recoverable inbox or quarantine workflow, with an atomic or safely replayable checkpoint protocol. That second model does not claim the business transitions are complete. For a transactional Kafka quarantine, readers must use read_committed when they rely on aborted output records remaining invisible. A Kafka transaction does not include an external database mutation.

A skip policy is a product decision

Some event types may be safely dropped after a bounded attempt because the next event is an authoritative full snapshot. For example, a location observation from five seconds ago might be obsolete after a new full location snapshot arrives. Even then, audit and data-completeness contracts may require tracking the omission.

Other event types are transitions, payments, balances, or inventory movements. A skipped delta can corrupt every following value. For those, quarantine blocks the dependent sequence until the gap is repaired or the source of truth issues a state replacement with explicit authority.

I would write the allowed responses by event type, rather than having one global maxRetries=5 then skip rule. That is how the system gets both availability and truthful ordering contracts.

For deterministic schema errors, record the incompatible schema ID and affected producers, then correct the deployment contract. The immediate rollback might be the producer, the consumer, or a compatibility bridge. A large DLQ rate after deployment is itself a release signal. Restarting consumers does not solve a producer that continues to emit invalid payloads.

Make redrive safe, not just possible

A repaired event should re-enter the correct logical sequence. If the consumer already moved beyond the original offset for unrelated keys, redrive into the head of a topic does not recreate its old position. The parked parcel state needs a pending version and an ordered resume protocol. A handler should reject conflicting replacements that reuse an event ID for a different meaning without explicit correction metadata.

Operators also need an audit trail: who quarantined it, why, whether it blocked a parcel, which repair was applied, and what the final consumer state became. A dashboard saying "DLQ count dropped to zero" says little if the main projection still has gaps.

The policy follows dependencies. Diagnose the error, preserve evidence, prevent unsafe transitions, and isolate genuinely independent work. A DLQ is one tool here, not the entire design.

A later incident raises a different issue. The consumer was offline long enough that the event no longer exists in Kafka. There is nothing left to quarantine or retry.