Security, Governance and Platform · Principal
The database is failing. Should the incident agent run its proposed migration?
The question
Interview question
Writes are timing out in two regions. An incident agent sees slow queries and proposes creating an index on a large payment table, then backfilling a new column and routing traffic to a replica. Its SQL passes syntax checks in staging. The on-call lead asks the agent to execute it because every minute costs money. What must be decided before any command runs? The metrics service is partly down and one region may be promoting a new writer.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
I would stop at the proposed action. Three operations have been bundled into one “fix,” and each has a different failure mode. An index build can consume I/O, wait on locks, and compete with the already struggling writer. A backfill can create more writes and replication lag. Routing to a replica may serve stale payment state or direct writes to a place that cannot accept them. Passing SQL syntax checks tells us almost nothing about the production conditions.
First establish the incident state from sources that still work. Which region has the writer lease, what are the observed write error modes, are reads and writes both affected, and is the replica caught up enough for its proposed use? Use database-native status, deployment history, and a direct low-rate probe where safe. If writer promotion is in progress, do not start DDL against an instance whose role or epoch may change under the operation. An agent can collect and compare evidence, but a metric gap remains a gap. The cost per minute does not turn missing topology into authorization.
The agent should output a change proposal a human database owner can actually inspect: exact SQL and target, current schema and index state, estimated table size, lock and I/O behavior for this engine and version, replication impact, deadline, stop condition, expected benefit, verification query, and a recovery path if it fails halfway. Split the index, backfill, and routing change into separate steps. Each step needs its own owner and observation window. If this is PostgreSQL, for example, CREATE INDEX CONCURRENTLY reduces some write blocking but can fail and leave an invalid index. It is not a magic “safe index” flag. A migration written for another engine cannot inherit those mechanics.
Who approves? The incident commander owns the incident priority, the database or service owner understands the data path, and the security or change authority defines who may perform privileged DDL. If a rehearsed emergency process exists, it can give a named human a scoped, time-limited grant for an exact command and target. The agent may run a dry check and prepare an execution plan, but its own diagnosis does not grant it production credentials. The existing emergency policy question covers loss of the policy service. This question is about whether a concrete proposed mutation is technically safe while the system is already degraded.
Suppose the index looks likely to help but metrics are unavailable. Could we execute on one region? Only if regions really have independent writers or a documented regional scope and we can observe at least the database's own progress and lock state. If both regions share the same source or replication topology, “one region” may still affect both. Use a short lock_timeout or equivalent where supported, rate and resource limits for the backfill, and a kill path that has been tested for the specific operation. Stopping a client command does not necessarily erase work already committed. If observability cannot confirm that the action is improving or harming the incident, choose a lower-risk mitigation such as shedding nonessential load or pausing the affected feature, when available.
The replica routing idea needs a separate correctness answer. Payment status is a stateful fact. A lagging read after a capture might tell an agent it has not happened and prompt another capture. Route only reads whose freshness contract allows it, or use a read-after-write token or primary read for consequential decisions. Do not rewire all traffic for a latency chart.
After any execution, record command identity, approver, writer epoch, observed database state before and after, and the exact range of rows changed. Test failure after partial backfill, lock timeout, writer promotion, and monitoring loss in a rehearsal. The best interview answer is not “always refuse emergency changes.” It is knowing exactly what can be executed safely, by whom, with what evidence, while the system is least forgiving.
Continue reading
Related questions
Read beyond the question
Explore more security, governance and platform
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →