Agent Architecture · Principal
The coding agent made CI green by changing the test. Do you merge it?
The question
Interview question
An agent is asked to fix a checkout bug where a coupon is incorrectly applied twice. Its patch changes the implementation, removes the failing assertion from an integration test, and reports that all tests pass. A developer says test edits are normal during a bug fix. Decide whether to accept this patch, and design a workflow that can scale to thousands of agent changes without trusting a green badge blindly. Then the old assertion turns out to encode a policy that changed yesterday.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
I would hold this patch and inspect the changed assertion against the intended behavior. Test edits can be right. Removing a test can also remove the only check of the bug. “CI is green” tells me what ran on a particular tree, not whether that tree implements the ticket. The agent can change both the candidate and part of its examiner, so those two things are not independent evidence.
First reconstruct the contract. For a coupon meant to apply once, choose a small example with the actual checkout inputs and expected final price. Check the current policy or product owner for stacking rules, the bug report's reproduction, and the prior test. Then run the old failing scenario against the patch, even if that requires restoring the assertion in a separate harness. A regression test should fail on the known bad version and pass on the proposed fix. That does not prove every case, but it shows the test can distinguish this bug from its absence. Also test the adjacent cases that could break under a one-line fix: zero-value coupons, retries, two different coupons, rounding, and an already discounted cart. Pick the cases that follow from the implementation, not a universal test list.
At scale, I would separate proposal from acceptance. The agent works in an isolated branch and returns a patch, base commit, changed tests, test commands and exits, and a plain description of the behavior it believes changed. A controlled CI runner supplies required checks from a protected location or trusted workflow. The agent may propose test changes, but it cannot edit the protected policy fixtures, required workflow, expected results of an independent acceptance suite, or the identity of the app reporting a required check. GitHub's documentation ties required checks to the latest relevant commit, and branch rules can require an expected status source. These controls only help if the acceptance checks themselves are meaningful and a privileged bypass is governed.
Diff inspection should flag weakened assertions, deleted tests, changed fixtures, new skips, changed CI paths, and broad changes outside the ticket. A flag triggers review, not an automatic verdict. Adding a better test may require deleting a wrong one. For high-impact paths such as money or access, a human owner should review the behavior and the acceptance evidence. For lower-risk changes, independent behavioral tests and sampled review can carry more of the load. Measure escaped regressions and the rate of rejected or reworked patches by change type. A high green-CI rate alone can reward agents for making CI easier to pass.
But the assertion may be based on yesterday's policy. Suppose coupons really may stack in one defined case now. I would not restore the old assertion as sacred. The policy owner must identify the effective time and the intended rule. Replace the outdated test with one that expresses the new behavior, keep a test for the still-forbidden double application, and compare the patch against both. A test's independence is about who controls the acceptance criterion for this change, not about freezing every test forever. If the ticket and current policy conflict, the agent should surface that conflict before merging a convenient interpretation.
There is a second trap after review: the agent's branch was green on Monday, but it merges into a different tree on Tuesday. Run required checks on the candidate merge tree, including the checkout integration path. A merge queue can test the latest base plus earlier queued changes. GitHub describes that behavior. The badge for the agent's earlier commit is evidence about that commit. It does not travel with an untested merge.
I would accept only a patch whose behavior matches the current contract and whose acceptance evidence was produced on the tree we will actually merge. A changed test is a reason to examine the claim, not proof of cheating and not proof of correctness.
Continue reading
Related questions
Read beyond the question
Explore more agent architecture
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →