Security, Governance and Platform · Principal
Half the fleet enforces the new policy and half the old one. When can rollout be called complete?
The question
Interview question
A customer revokes partner access to its documents. The policy service publishes revision 42. Half the serving nodes have it. The others still enforce revision 41 and can allow a partner read. A dashboard says “policy deployed” because the control plane published the bundle. When may the product tell the customer that no new partner read will be accepted?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Only when the acceptance path has a real cutoff. Publishing revision 42 is not the same as every active node enforcing it. OPA status reports active bundle revisions, and decision logs can include the bundle revision used for an evaluation. Those are useful observations. They do not automatically fence a node that is still serving with revision 41.
I would define an effective policy epoch for this revocation and a gateway rule: a partner read for this customer cannot proceed unless the authorization decision comes from a policy view at least as new as epoch 42. That can be implemented through an online authoritative check, a propagated minimum epoch that every serving path enforces, or by draining and excluding nodes that have not activated revision 42. The design depends on availability and latency targets, but there must be one place that prevents the old node from accepting the forbidden operation. A node that missed the update cannot simply keep accepting because its local policy says yes.
Now ask when we acknowledge the revocation. If the promise is immediate after acknowledgement, first establish the gate or remove stale nodes from all routes, including retry and background job routes, then acknowledge. In-flight reads already authorized or submitted need a written rule. If a response containing private data has already crossed the disclosure boundary, a later epoch cannot pull it back. If a request was admitted before the cutoff but has not returned bytes, the product may need to recheck at disclosure. The exact boundary should be chosen from the customer contract and made observable. Do not imply that all machines share an instantaneous wall-clock moment.
For the rollout, record the intended revision, active revision by node and region, last decision under revision 41, decision outcome, principal and resource scope, and when a node was removed from traffic. Keep enough audit detail to determine whether any disallowed read was accepted after the promised cutoff, while protecting sensitive inputs in logs. A stale node that was offline during rollout must come back behind the same minimum-epoch gate. Health checks need to distinguish process alive from policy ready. Count the active fleet and the policy age distribution, not just successful bundle downloads.
If the policy backend is unreachable, decide whether a protected read waits or denies. Falling back to revision 41 violates the new revocation promise. For less sensitive policy changes, the product may accept bounded staleness, but state and measure that contract rather than pretending every change has the same urgency. If the interviewer says waiting for all nodes will hang forever because one node is dead, do not wait for that node. Prove it cannot serve through routing and credentials, and require it to catch up before rejoining.
The rollout is complete for this customer when every path that could disclose the documents is either enforcing the required epoch or fenced from accepting the read, and the last old-policy acceptance is before the committed cutoff. The batch passed policy at 9 AM. Can it publish results after the rule changes at noon? asks whether a batch can publish after the policy changes. This page asks how a distributed serving fleet makes that commit-time check mean the same thing across nodes.
Continue reading
Related questions
Read beyond the question
Explore more security, governance and platform
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →