Agent Architecture · Staff
The provider fails after a tool result. What can the next model actually continue?
The question
Interview question
An agent has looked up a customer invoice and received a verified tool result. Before it can answer, its model provider becomes unavailable. The gateway sends the conversation to another provider. The new model says the invoice is paid and proposes a refund, although the old model was investigating whether a payment was still pending. Design the handoff. The first provider stored the conversation under an opaque response ID, and its private reasoning cannot be exported.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
A provider response ID is not a portable checkpoint. It names state held by that provider under its own API contract. The next model can only work from material our runtime is allowed to give it. We should not pretend to transfer the first model's hidden reasoning or exact continuation. We are starting a new reasoning attempt from a known boundary.
That boundary is after the invoice lookup has completed and before any new effect. Persist a durable task record with the user's goal, authenticated account and permitted scope, the tool request and result under stable application IDs, the invoice source revision, any policy and approval state, and what has already been shown to the user. Record that payment status was under investigation, not that the first model concluded it was paid. The handoff context can be short, but it has to distinguish verified facts from the model's hypothesis and from unknown provider outcomes. A tool result formatted as prose is especially risky if it contains instructions from an external system. Reconstruct the relevant fields through a typed adapter and retain a source link for inspection.
The runtime checks that the fallback model may receive this tenant's invoice data in its approved region and retention terms, supports the needed tools and output contract, and fits the task's quality gate. If no route meets those conditions, pause and tell the user. A fast answer from an unapproved route is not a recovery. Provider-native tool call IDs also must not become business operation IDs. OpenAI's function calling guide and Claude's tool use guide show different message and tool-result forms. Our application ledger maps either form to the same durable tool invocation and effect state.
For this invoice, the new prompt should include the actual payment status field, its timestamp and source, not the prior model's unverified sentence. If the invoice service says pending, the new model can explain that and ask for the next permitted check. If status is ambiguous because one provider response was lost, the runtime has to reconcile the invoice or payment operation before allowing a refund. This question begins after a completed read result. The earlier gateway and stream questions already cover interrupted proposals and unknown writes. Mixing those states is how a provider switch accidentally duplicates an effect.
Now suppose the fallback model has a smaller context window and the original conversation contains a customer exception in an attachment. A summary that drops the exception is not a safe compression. Select the governing facts and evidence by task, include the attachment revision if this route is permitted to access it, and check that its necessary clauses fit. Otherwise request a human review or wait for the original provider. A capability matrix tells us what the route can accept, but it cannot prove that a compressed context preserved the right decision boundary.
What if the user already saw “I think this invoice is paid” before failover? Keep that visible as an interrupted provisional claim and correct it if the verified status differs. A fresh model answer cannot silently replace what was displayed. Test fallback after a clean read, after a partial user-visible sentence, with a missing attachment, under a residency restriction, and after an effect whose outcome is unknown. Measure coherent verified completion and repeated or unauthorized actions, not only failover success rate.
Continue reading
Related questions
Read beyond the question
Explore more agent architecture
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →