Distributed Reliability · Principal
The binary rolled back. Can the old version read the data the new version already wrote?
The question
Interview question
A new reservation service version writes a new `payment_state` value, `partially_captured`, and starts moving data from `amount` to `amount_minor_units`. The canary exposes a bug and the deployment controller restores the old binary. Old instances now reject the new enum and may read zero from the old amount column. The rollout dashboard says “rollback complete.” Design a release and rollback plan. Some new writes have already triggered external payment captures.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The controller restored an image. It did not restore the database or undo payment effects. If the old binary cannot read records written by the canary, the rollback can widen the incident. Before reaching for the old image, I would stop new exposure, identify which records were written in the new format, and decide which serving version can safely process them. A narrow forward fix or a compatibility bridge may be safer than sending every request to code that misreads money.
Design the release so the previous binary remains compatible with the new writes until the rollback window closes. Add amount_minor_units while keeping amount, populate it from an authoritative amount with explicit currency and rounding rules, and verify the backfill. During the transition, make one value authoritative and derive the other on every write, ideally inside the same transactional boundary where both columns live. Do not run two independent writers and hope eventual convergence preserves payment amounts. New readers can compare fields and surface mismatches. Old readers continue to see the old field. Only after every old reader, delayed job, and rollback target is gone should the old field be removed. This is the expand, migrate, contract idea in Parallel Change, applied to actual monetary data rather than just column names.
The enum requires the same thought. If old code rejects partially_captured, the new code must not produce that state while old code may still run, unless all reads of that state are routed to capable code. A feature flag on the write behavior can let us deploy parsing support to old and new versions first, then enable the new state later. If the business needs the new state immediately, the rollback target is a newly built compatible version, not the original binary. Unknown enum values should not be silently treated as paid or unpaid. A reservation in an unknown payment state must stop unsafe fulfillment and be reconciled.
For the current incident, enumerate affected reservation IDs and their exact payment operation IDs. The provider receipt, not the rolled-back process, establishes whether capture occurred. Do not replay a capture because the old service sees an unfamiliar state or a timeout. Put affected records in a safe processing lane, query the provider, and repair internal state with auditable transitions. An already captured payment cannot be “rolled back” by a database restore. A refund, if needed, is a new authorized business action. Keep reads truthful for users even if fulfillment is paused.
Kubernetes Deployment documentation describes how a rollout can replace and restore ReplicaSets. That machinery says nothing about data compatibility. Before release I would test the version matrix: old reader on new writes, new reader on old rows, mixed-version concurrent writes, queued messages from both producers, backfill retry, canary rollback, and provider response arriving late. Tag records written by the new format or capture a migration watermark so we can locate the affected set without assuming the canary traffic percentage equals the record percentage.
If an interviewer insists that all migrations must be reversible, I would push back on the word. Some effects, including external captures and information already sent to clients, cannot be undone by a down migration. The goal is a recoverable transition with a truthful forward path, a safe serving version, and a finite window where both shapes coexist. “Rollback complete” should mean users can safely read and act on the data, not just that the pod image name changed.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →