The interview question. A worker acquires a lease from a reliable coordinator, begins generating an export, and pauses for 90 seconds. Its lease expires. Another worker obtains the lease and writes a newer export into shared storage. The old worker wakes up and writes its older export over the new one. Both workers correctly used the lock API. Prevent the stale write, identify the component that must enforce the protection, and explain what happens when the destination cannot validate ownership generations.
Treat the coordinator's lease service as functioning correctly. The bug is at the boundary between a claim to ownership and the external side effect. Even a perfectly configured coordinator cannot stop a paused process from resuming its already-running code; the destination must enforce which effects it accepts.
Think first
Worker A gets lease epoch 81 and pauses. Worker B gets 82, writes new output, then A resumes and tries to overwrite it. Explain why a pre-write lease check is insufficient, where token comparison must run, and what the guarantee becomes if A arrives before B’s first write.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
A lease tells you who should work, not who is physically stopped
A obtains lease generation 81. It begins a long transformation and then pauses during garbage collection or an operating-system stall. The coordinator sees no renewal and expires A's lease. B acquires generation 82 and finishes the export. Nothing unusual has happened at the coordinator.
Now A resumes. Its in-memory variable still says leaseAcquired = true. It may never have received an expiration notification. Even if it does receive the notice eventually, a write request already in flight cannot necessarily be recalled.
Old worker A resumes with fencing token 81 after B has committed token 82, and the storage resource rejects A
Checking the lease just before writing narrows the gap but does not close it. The worker can check successfully, pause, lose the lease, and only then send the write. That is another check-then-act race separated by an unpredictable delay.
Make the resource reject stale authority
The coordinator issues a monotonically increasing fencing token whenever ownership is acquired. A holds token 81. B holds token 82. Every write to the protected resource includes its token, and the resource checks it as part of accepting or rejecting the update.
In a simple whole-resource example, storage maintains the greatest generation it has accepted. Once a write with generation 82 commits, a later write carrying 81 is rejected. The older process is still alive, but it no longer has authority to change this resource. This is the core fencing-token technique described by Martin Kleppmann.
The check must be atomic with the protected mutation. A separate request that asks "what is the current generation?" and then writes a file later can race in the same way as the earlier lease check.
The generation should be scoped to the resource or ownership domain. If different resources independently rotate owners, one global comparison across unrelated exports is unnecessary and can create false rejections. The authority issuing the token must itself produce ordered generations that survive its supported failures.
One subtle case: the old write arrives first
Suppose A's delayed write with token 81 reaches storage after its lease has expired but before B's token-82 write reaches storage. If the storage resource only remembers the greatest token previously seen, it may still accept A's request at that instant.
So a high-watermark check is not an unconditional promise that no stale operation can occur after the coordinator awards a new lease. It prevents an older generation from overwriting or following a newer generation that the resource has already accepted. To reject every old operation immediately at lease transfer, the destination also needs an authoritative handoff or validation of the current generation before accepting effects.
That extra boundary matters when a single stale write itself is harmful. For example, sending a physical shipment twice cannot be repaired just because the second writer arrives later. An interview answer should name whether the business needs monotonic output versions or strict exclusive effects at the moment ownership changes.
For our replaceable export file, an effective protocol is to stage each output under a generation-specific immutable object name, then conditionally update the visible pointer under a generation check. The pointer must never move backward from version 82 to 81. Publishing a complete object before moving the pointer also avoids exposing half-written files.
This approach is stronger than overwriting a shared path in place. It separates producing bytes from deciding which version is visible. If A finishes after B, its old staged object may simply become garbage collected later.
What if storage cannot check a token?
Then the coordinator alone cannot provide the claimed safety. There may be another authority in the path: a database row with conditional writes, a fenced publishing service, a versioned object pointer, or a workflow that verifies completion before exposing it. But routing through that component is essential. An unprotected direct-write path defeats the design.
Sometimes the destination offers idempotency keys but not fencing. Those solve a different problem. A stable key can suppress repeating the same operation. Fencing decides whether an older owner may perform a different operation after a newer owner exists. They may both be necessary.
Retries must retain the generation appropriate to their original work. A stale worker must not obtain a fresh token simply to make its old output eligible. A legitimate new worker should produce a new attempt only after checking the intended input version and business state.
For a real test, suspend A after it acquires the lease, let B acquire a higher token and publish, then resume A. Verify that the visible output cannot regress. Also test the moment before B's publication, because that exposes whether the design provides strict lease exclusivity or only ordered publication.
The interview question is not "Which distributed lock library would you use?" It is which resource can still be changed by an obsolete worker, and what will that resource reject?
The next question returns to the Raft leader itself. It has a complete local state machine. Can it serve a linearizable read without first proving it is still the leader?
Go deeper
Comparing fencing strategies
One design stores the largest committed generation beside the protected value. Another uses a conditional pointer swap against the expected current version. A third uses a destination API that recognizes ownership epochs. The details differ, but none can be bypassed by sending an unversioned write.
Stale token and clock time are not equivalent
Fencing tokens express ordered authority. Comparing wall clocks or checking that an operation started before an expiration time is not an adequate replacement when pauses and network delays are unbounded.
Hard external effects
For sending a payment or triggering a machine, use a destination that supports an operation identity and, where necessary, authorization by the current owner. If it cannot, reconcile the unknown result instead of promising a guarantee the system cannot enforce.
Check your reasoning
Explain the check-then-act gap caused by unbounded pauses and in-flight network requests.
Use monotonically increasing fencing generations with a destination-side atomic check.
Distinguish preventing regression after epoch 82 is seen from excluding every epoch-81 side effect immediately at transfer.
Explain immutable generation-specific output with conditional publication or an equivalent fenced mutation.
Describe failure when the destination cannot validate epochs, and distinguish operation deduplication from ownership fencing.
Common mistake: Using a time-based lease check in the worker as proof that its external write is still authorized.
Read the answer explanation
Reject token 81 as lower than the accepted generation 82. — The destination must enforce versioned authority, not rely on the stale worker’s local lease state.
Apply it elsewhere
Two agents generate competing deployment plans. A stale agent finishes after a new plan was approved. How do you prevent publishing the obsolete plan?
Compare your approach
Stage outputs immutably and conditionally publish only under the current approved generation, with authority checked at the publication resource.
One more constraint
If token 81 reaches storage before token 82 is ever presented, is a simple largest-seen-token check enough for strict lease exclusivity?
Quick revision
A lease cannot stop code already executing. A worker may pause past lease expiry, then resume with obsolete local state or an in-flight write. Give successive owners increasing fencing tokens, and have the destination enforce them atomically with the mutation. After token 82 is accepted, token 81 must not overwrite it. A largest-seen-token check does not necessarily reject the older attempt if it arrives before the destination has seen 82, so strict exclusivity needs an authority handoff or stronger destination validation. For replaceable output, stage immutable versions and conditionally publish the current pointer.
