Data and Knowledge Systems · Principal
We retained the Kafka topic. Why can't the agent reconstruct last year's policy changes?
The question
Interview question
What does “retained” mean for this topic? With log compaction, Kafka keeps a recent value for each key so a consumer can rebuild *current state*. Older values for that key can be removed during background cleaning. A tombstone can eventually remove the key from that reconstructed state too. [Kafka's design documentation](https://kafka.apache.org/43/design/design/) describes compaction as a latest-value-per-key mechanism. It does not promise to preserve every change event forever just because the topic still exists and its beginning offset can be read.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Imagine policy refund-limit changed from 100 to 200 and then 150. A newly started consumer of a compacted topic may see only 150. That is enough to build the latest policy table. It cannot answer “who raised it to 200 last March, and what did the agent know that day?” A long-lived consumer that processed every event earlier might have written its own history, but replay from the compacted topic alone cannot recreate what compaction removed. Even a non-compacted topic is not automatically a permanent audit archive. Its time and size retention settings still matter.
I would inspect cleanup.policy, retention settings, compaction and tombstone age, key design and actual oldest records. If each update has a unique event ID as its key, compaction behaves differently, but retention and audit integrity still need a deliberate design. For a defensible history, publish immutable change events with actor, effective time, recorded time, old and new version or a stable reference, then retain them under a governed policy. Materialize current state separately. If policy updates and event publication can diverge, use a transactional outbox or equivalent recovery path, and reconcile the current policy against the audit stream.
This matters to an AI agent because asking it for a historical explanation requires evidence from that historical moment. A current compacted policy topic can tell the agent what rule applies now. It cannot safely supply a missing chronology. If the history is gone, the honest answer is that the evidence is unavailable, not a plausible narrative inferred from today's value. The audit log says approved. Can you prove what the reviewer saw? asks whether an approval log proves what a reviewer saw. Here the log chosen for source replay was designed to discard intermediate values.
Continue reading
Related questions
Read beyond the question
Explore more data and knowledge systems
Follow another question in this area, or search the complete Question Library.
Browse this area →Browse Question Library →