Choose which failure you want
A consumer needs two things to happen. It must perform the business effect and it must record that the message is finished. Putting them in either order still leaves a crash window.
For an ordinary at-least-once consumer, I prefer to complete the business transaction before advancing the offset. The second failure is manageable when the handler recognizes work it already completed. The first risks silently losing work.
Say Kafka delivered offset 420. The warehouse reserves stock and commits, but its saved restart position is still 420 rather than 421. After a crash, another consumer resumes at 420. Kafka is behaving as expected. Its committed offset records consumer progress, not what happened inside the warehouse database.
Store proof of the effect with the effect
I would keep a processing receipt in the warehouse database. Its unique identity is scoped to the tenant, the logical handler, and the event ID:
tenant_id + handler_name + event_id
The handler is a stable purpose such as inventory-reservation, not the current worker's hostname. Analytics and inventory may both process the same event, so they should not suppress each other. Within one warehouse transaction, try to claim the receipt, conditionally reserve stock, and record the outcome. A repeated delivery finds the committed receipt and returns without reserving again.
The receipt must not be committed in advance of the business effect. Otherwise a crash could leave proof of completion with nothing completed. The database's uniqueness constraint also matters when two workers briefly overlap during a rebalance. A process-local set of seen IDs cannot arbitrate that race.
An insufficient-stock decision can be a terminal outcome too. Store it so redelivery does not unexpectedly change yesterday's rejection into today's reservation. A transient database error is different. Roll back and allow recovery to retry it.
There is a second invariant that the receipt does not cover. Two different orders can both race to reserve the last available unit. Deduplicating each event does nothing to prevent overselling across orders. The inventory decrement still needs a conditional update or another transactionally enforced allocation rule.
For additional protection, the fulfillment record should have a stable business identity such as (tenant_id, order_id, fulfillment_version). That catches a producer that mistakenly creates two event IDs for the same fulfillment intent. The event receipt and business uniqueness rule protect different boundaries.
Committing offsets safely
After the database transaction resolves, the consumer may save its broker progress. If it crashes first, delivery repeats and the receipt makes the replay harmless. This gives at most one committed effect per retained identity, provided the local transaction and its records survive. Eventual completion still depends on retries, capacity and a recoverable broker history.
If you add parallel processing within a partition, the offset question becomes more interesting. Record 420 finishes. Record 421 is slow. Record 422 finishes. Saving 423 as the restart position would skip 421 after a crash. Advance only past the contiguous prefix of durably completed records, taking account of the actual records delivered rather than assuming all integer offsets contain application data.
You can avoid that complexity initially by processing one partition serially. Or you can store the processed position together with the database effects and explicitly manage ownership when partitions move between workers. Neither choice magically deduplicates the same business event when it appears under another offset or in another cluster.
The interviewer changes one sentence
The warehouse now calls a third-party fulfillment API. Your database stores only workflow state.
Our local transaction cannot roll back the provider's reservation. The provider may commit it, the response may disappear, and our transaction may then abort. Treating the call as a normal function inside a database transaction conceals that difference.
I would first persist a durable local fulfillment operation with a stable provider operation ID. A worker invokes the provider and stores the result. If the call times out, the operation becomes UNKNOWN, not FAILED. Recovery looks up the provider's outcome or retries with the same provider identity, but only if the provider's contract supports that safely.
The consumer can advance its broker offset after it has durably transferred responsibility to that local workflow, even if the provider has not yet finished. The workflow must have its own recovery worker. The user-visible status remains pending or unknown until the provider result is established.
If the provider offers no stable idempotency identity and no reliable lookup for an uncertain call, you cannot guarantee both that the reservation will eventually happen and that it will never happen twice. Another retry risks duplication. Refusing to retry risks omission. That limitation comes from the unknown external outcome, not the queue configuration.
The same reasoning applies to webhook receivers and payment callbacks. The transport acknowledgment is a progress signal. The business outcome needs its own durable record.
This warehouse keeps generic processing receipts for 48 hours. An operator now proposes replaying 90 days of order events to rebuild a dashboard. Will the old receipt table protect the warehouse from those events?