Data and Knowledge Systems · Staff
The incident agent read today's code. What was actually running during Tuesday's outage?
The question
Interview question
An incident agent explains Tuesday's payment outage by citing a bug on the default branch. The bug was fixed Wednesday, and the affected service was running two builds across three regions on Tuesday. A feature flag changed during the incident and one customer used a tenant-specific configuration. Design the evidence path for “what code caused this request to fail.” One region has incomplete traces.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The default branch tells us what is in source now. Even the release branch does not prove which binary handled a given request on Tuesday. Start from the failed request or affected operation, its time, region, service instance or trace, and the artifact digest that instance reported. Map the artifact back to a build and exact source commit, including generated code and dependencies where they can change behavior. Join that to the effective flag and tenant configuration at the time the request was evaluated. A flag snapshot from the end of Tuesday is not proof of what the request saw earlier.
I would build a historical deployment ledger: immutable build artifact, commit and build inputs, rollout event, region and instance, routing assignment, configuration revision, flag evaluation or version, and time bounds. Its clock fields need care. “Deployed at 10:00” may mean the control plane requested rollout, not that every instance served traffic then. Keep observed serving intervals from telemetry and load balancer membership, with uncertainty where those observations are incomplete. OpenTelemetry's resource conventions include service.version as a way to identify a running service version, but instrumenting that label is our job and it is only one join key. It does not reconstruct a tenant flag evaluation by itself.
The answer should distinguish three claims. “This source line exists in commit C” is a code fact. “Build B from C served this request” is a deployment and routing fact. “This line caused the payment failure” is a causal claim that needs a mechanism and corroboration. Reproduce the failing input against B and its effective configuration, inspect the first failing span and downstream error, compare unaffected requests on the other build, and check whether a dependency changed. If a rollback reduced errors, that supports the hypothesis but does not identify the exact line on its own. The agent should cite the evidence for each step and be able to say where a join is missing.
What about the region with missing traces? Query its deployment records, gateway logs, metrics with build labels, and ticket timestamps. The confidence should narrow: “Two observed failed requests were served by B in region one. Region three's build attribution is unknown for this interval.” Do not silently generalize from the well-instrumented regions. If the customer traffic could have reached both builds, report both candidates and the routing uncertainty. Historical config may be reconstructed from an append-only change log if the actual evaluated value was not logged. Mark it inferred, particularly when propagation was asynchronous.
The feature flag changed halfway through the outage. If it is evaluated per request, record the decision and flag version on the request trace where privacy and cost allow. If it is cached on a worker, the control-plane change time is not the effective time. The same goes for tenant settings. A time join must account for cache refresh and clock skew, or it will manufacture a neat but false narrative. I would sample and retain enough trace context for incident classes where this matters, then document what sampling cannot prove.
Now suppose the bug on today's default branch looks identical to an older bug in B. The agent may use current code to suggest a search path, but it must cite the historical commit for the explanation and verify the behavior under the old dependencies and flag state. This extends the code impact graph questions in Which code actually consumes a changed event? and Can the assistant say no consumers exist? from possible consumers to a specific runtime history. A useful incident assistant says what is known about the actual serving path, not just what code it found quickly.
Continue reading
Related questions
Read beyond the question
Explore more data and knowledge systems
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →