Five dashboards disagree. What do you do in the first ten minutes?

Interview question. An online marketplace's checkout success rate drops sharply. API latency rises in one region, database CPU is flat, a dependency error counter is normal, and a new release was deployed twenty minutes ago. You have not found the root cause. Run the first ten minutes of a Principal-level incident investigation without accidentally making the incident worse.

The instinct is to open every dashboard and search until an explanation appears. That can take an hour, and meanwhile customers continue failing. I want a different first ten minutes: establish the customer impact, contain the damage if possible, and collect enough discriminating evidence to avoid making three speculative changes at once. Root cause is important. Stopping the harm is often more urgent.

Minute zero is about making the event real. Check whether checkout failures are verified by an external probe and by application outcomes, not simply inferred from one graph. Define the incident boundary: which region, operation, customer cohort and start time? Are successful checkouts genuinely completing payment and confirmation, or only returning an acceptance response? Record the first clear symptom and a narrow error-budget or business-impact estimate. Page the appropriate service owner and declare the incident if the escalation criteria are met. One person should coordinate while others investigate. A confusing dashboard disagreement is not a reason to avoid ownership.

Next, freeze unrelated changes. We have a release at 14:10 and symptoms at 14:22. That temporal relationship makes the release a strong suspect, but not a proven cause. Record the exact build, config and migration versions deployed, the rollout percentages, plus any concurrent traffic or dependency change. Compare affected and unaffected groups. If the new version serves only region east, then regional error concentration supports the hypothesis but cannot tell us whether east itself is failing. We need an intervention that distinguishes the alternatives.

Here's how I'd write the hypothesis table on the incident channel. H1: the new release changes a database query and slows only baskets with many items. Prediction: p99 and slow-query counts worsen for new-version large baskets, not for the stable version. H2: one external tax provider is intermittently slow. Prediction: slow requests correlate with that provider's destinations, and bypassing or falling back under an approved policy improves customer outcomes. H3: regional queueing has risen before requests enter application code. Prediction: edge duration increases without a matching increase in application span duration, and the pattern exists on both builds in the region.

A good experiment changes one thing and has an expected observable result. Pause rollout before collecting more logs. If rollback is independently safe, roll back the new version for the affected cohort while preserving an evidence sample. Watch the eligible checkout success rate, end-to-end latency and payment reconciliation for improvement. If errors fall immediately, the release becomes a more credible cause. It still might be an interaction between the release and a particular dependency, so write the finding accurately. If nothing changes, do not declare the rollback itself a failure and blindly redeploy. Update the hypothesis list.

What if we cannot safely roll back? A database migration may have removed a column required by the previous binary. Or a payment side effect cannot be undone by changing application code. In that case, use a feature flag, targeted rate limit, or traffic reduction with a documented invariant. Disable a newly introduced optional recommendation step only if checkout correctness remains intact. Never test by sending real double captures to a card processor. Use synthetic accounts or a controlled sandbox and explicitly separate observed production behavior from test data.

The dashboard disagreement is useful evidence if we know what each instrument observes. Flat database CPU may hide lock waits, connection-pool starvation, or a bad plan that increases I/O. A normal dependency error counter may mean timeouts are reported at the gateway rather than on the client instrumentation. An API latency increase in one region may reflect cross-region calls or extra retries. Every hypothesis should ask, "What metric would look different if this were true?" and "Could this measurement be stale or sampled?"

I would run one tightly bounded query over request IDs that failed during the last five minutes, grouping by route, version, region and outcome. Then compare the matching successes. Look for the first differing critical-path step, not the loudest log line. If trace sampling is sparse, use aggregate version-specific distributions and selected request logs instead. Watch the telemetry pipeline's freshness so an empty trace query is not mistaken for the absence of a traceable failure.

Keep an incident timeline with observation, hypothesis, intervention and result as separate fields. At 14:24, for example, "east checkout errors 8%" is an observation. "The release caused errors" is a hypothesis. "Pause rollout and route 5% of east traffic back to stable" is an intervention. "Errors on that cohort returned to baseline within three minutes" is evidence. This discipline stops a narrative from hardening before the experiment has earned it.

At minute ten, either we have a mitigation, a named next experiment, or an escalation because the remaining options require authority or risk beyond the on-call's remit. We also establish customer communication, a single incident commander, and a short update cadence. Resist the temptation to spend the first hour rewriting the monitoring dashboard. Reliability teams earn trust when they reduce harm and say clearly what they know, what they suspect and what they have not yet verified.

The interview is assessing operational judgment under uncertainty. A good answer is not an exotic root cause guessed instantly. It is a process that limits the incident blast radius, preserves business invariants, tests falsifiable hypotheses, and makes it easier for the next engineer to join without repeating every failed experiment.