Distributed Reliability · Principal
The outbox row says sent. Why did no index event reach Kafka?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Check when the relay marks a row sent. In a custom polling outbox, a worker may read the row, enqueue a producer send, then mark the row complete before the broker acknowledges the write. If the send fails or the worker dies, the database now says published while Kafka has no record. The outbox protected the transaction that created the business row and outbox row. It did not make the later broker publication part of that same database transaction. Debezium's outbox pattern explanation describes the need to tolerate duplicate delivery in an at-least-once pipeline.
For a polling relay, wait for a confirmed broker result before advancing durable publication state. Do not confuse “accepted into the producer's local buffer” with the broker's acknowledgment. Use a stable event ID and partition key, record the broker outcome and monitor rows stuck in pending or in-flight states. If the process crashes after Kafka accepts the event but before the database records success, the relay will publish again. That is the unavoidable other side of the gap without a shared transaction. Consumers need idempotent handling keyed by event ID and business revision.
A CDC-based outbox relay has a different cursor and acknowledgment mechanism. It reads committed outbox inserts from a database log and publishes them, so there may be no per-row sent flag at all. Do not diagnose a Debezium connector by looking for a field it does not use. The same end-to-end questions still apply: what position can it restart from, when is progress committed, and how do consumers handle duplicates? Inspect the actual relay implementation before choosing the fix.
I would inject failure at four points: before producer send, after local enqueue, after broker acknowledgment and before recording publication success. For each, check whether the index event is absent, delivered once or duplicated, and whether recovery converges. The index event arrived, but the document transaction rolled back. What should search believe? covers an event emitted before the source document transaction committed. This case begins with a valid committed outbox row and loses it in the publication handoff.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or search the complete Question Library.
Browse this area →Browse Question Library →