Interview question. Your parcel-event topic retains data for seven days. A reporting consumer group is offline for eight. When it returns, its stored offset points to data Kafka has already removed. Operations resets it to earliest, and processing resumes. Can we tell the reporting team the missing day has been recovered? Design retention and recovery so this failure does not silently delete required business history.
Assume this is an ordinary delete-retained event topic, not a compacted changelog. No separate archive has been guaranteed in the problem. The report requires complete monthly history.
Think first
A reporting consumer has been stopped eight days while its topic retains seven days of events and the group offset falls before the earliest retained offset. Define what operators may claim, how the report should behave, and the archive and replay policy required for complete monthly history.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
An offset is a position, not a reservation
A Kafka consumer group stores progress as offsets. That progress tells the group where to resume. It does not tell the broker to retain all older data until the group reads it. A log's retention policy is a separate contract.
When Kafka removes an eligible log segment, an old offset can fall before the partition's earliest available offset. Resetting the consumer to earliest means starting at the earliest remaining record. It does not recreate what was deleted. Choosing latest would skip still more history. The auto.offset.reset behavior needs to be explicit and visible whenever a stored position is no longer valid.
A committed consumer position falls behind the earliest retained offset after an eight-day outage on a seven-day topic
Seven days is not a promise of deletion at exactly 168 hours. Kafka cleans log segments, and size-based retention can also make records disappear before the time window one expected if the effective configuration and workload permit it. The Kafka 4.1 topic configuration documents retention.ms, retention.bytes, and segment-level deletion.
For this scenario, the safe statement is that required history is no longer available from the topic alone. We do not know its exact missing set without examining the retained segment ranges and an independent record of what was published.
Detect the loss before the reader quietly restarts
For critical consumers I would not rely on a default reset that can silently turn lost history into a green dashboard. The restart path should compare committed offsets with each partition's beginning and end offsets. If the committed offset is out of range, mark the input incomplete and stop publishing results that claim full coverage. Operators must see the affected partition ranges and a clear recovery state.
Lag in record counts is only one alert. The system should also monitor the age of the oldest required unconsumed event, retained-history headroom, consumer downtime, and whether the event source remains reproducible. A busy topic can still be close to expiring required history.
Define an actual retention requirement from the longest credible interruption, detection delay, repair time, and intentional replay horizon, with some margin. If disaster recovery needs a 30-day reconstruction window, a seven-day topic is not meeting that contract just because normal consumers are usually caught up.
Archive and replay must be designed before the outage
There are several possible durable sources. A long-retention event archive in object storage can hold immutable records with stable IDs, schema versions, ordering metadata and integrity checks. Kafka tiered storage is another option when correctly configured and validated for the broker version and operational environment. An authoritative database might support reconstruction of current state, but not necessarily every historical transition required by analytics.
Do not casually substitute a compacted changelog for full event history. Compaction can retain the latest value per key while removing intermediate values. That is useful for reconstructing a table-like state, but not sufficient for replaying every historical parcel scan. Delete tombstones also have separate retention considerations. The Kafka design documentation distinguishes log retention and compaction semantics.
A deliberate recovery might restore the missing ranges from an archive into a controlled replay consumer, then catch up with the live topic. Capture partition and archive boundaries, compare source ranges and event identities, and build a separate version of the reporting output. Verify completeness before changing the reader pointer. The replay path must isolate reporting updates from fulfillment side effects. The issue here is whether the source bytes still exist; safe replay cannot recover a history that no durable source retained.
If there was no archive and the historical source cannot reconstruct the deleted events, the report has an irreparable gap. Mark it as incomplete. Estimates are not recovered transactions.
The capacity calculation belongs in the retention decision
Suppose the topic receives 2,000 records each second with 500 bytes of average retained payload. Seven days represents roughly 1.21 billion records and 605 GB of raw payload, before indexes, broker overhead and compression. Three full replicas would hold roughly 1.81 TB of that raw payload under simplified assumptions. A 30-day window would be more than four times as large.
That explains why keeping everything on broker disks may not be the right choice. It does not justify removing the only recoverable copy of data that the product contract requires. Use a cheaper storage tier or explicitly reduce the promised replay window. Test restore throughput as well as storage capacity. An archive that takes six weeks to restore cannot satisfy a two-day recovery objective.
A useful followup is what happens when the oldest required event becomes dangerously close to expiry. The system can alert, prioritize catch-up work, increase source retention if safe and feasible, or protect the required range through a validated archive. It should not simply scale app servers without checking the partition and downstream bottlenecks.
A broker may be healthy, a consumer may be running, and the reconstructed result may still be wrong. Successful polling proves a consumer can run. It does not prove the history is complete.
The next failure is not a slow consumer or expired data. A new tenant feature triples producer traffic and suddenly every team notices higher latency. Whose load is allowed to consume shared broker capacity?
Go deeper
Kafka evaluates delete retention with topic policies, log segments and retention checks. retention.bytes is a per-partition budget and can change the effective horizon under skew. Remote tiered storage has its own enablement, lifecycle and local-versus-remote retention controls. For an archive, design checksums, schema retention, access policy and a tested gap-free handoff to the live log. Verify ranges using original partition identities and source offsets when possible. A compaction policy that keeps one value per key is not a substitute for a complete audit stream even if its records are older than the delete-retention window.
Check your reasoning
Explain consumer offsets do not pin retention and earliest remaining is not earliest published.
Do not silently reset and declare historical completeness when offsets are out of range.
Distinguish delete retention from compacted latest-state reconstruction.
Specify validated archive or tiered storage, per-partition replay boundaries and completeness checks before cutover.
Estimate storage and prove recovery-time objectives with replay throughput, not retention size alone.
Common mistake: Believing auto.offset.reset=earliest restores deleted data, or assuming compaction preserves every historical transition.
Read the answer explanation
The consumer can begin at the earliest remaining offset, but the missing range still requires another source. — An offset tracks progress, not a reservation on deleted log segments.
Apply it elsewhere
An audit consumer needs one year of payment events, while hot broker disks only retain two weeks. What must change?
Compare your approach
Use a complete immutable archive or supported tiered retention with integrity, discovery, schema and replay testing, not just a longer client timeout.
One more constraint
How would you distinguish a gap caused by retention from an upstream producer that never published the event?
Quick revision
Consumer offsets record position, not a broker promise to retain every unread record. Resetting to earliest after retention loss begins at the earliest remaining record, not the beginning of history. Alert on retention headroom and treat an offset gap as incomplete input. Reconstruct required events only from a validated archive or durable tier with enough history. Compacted logs often preserve latest key state, not every event needed for audit.
