Distributed Reliability · Staff
One malformed agent job keeps coming back from the queue
The question
Interview question
An agent worker reads jobs from an at-least-once queue. One job contains a malformed tool payload. It fails parsing, becomes visible again, and is delivered repeatedly. Other jobs in its ordering group stop making progress. The team proposes retrying forever because dropping a user task is unacceptable. What should happen?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The queue has done its job by delivering the message. Retrying the same bytes through the same parser cannot repair them. Infinite retry is not preservation. It can block later work, burn capacity, and hide the user's task in a loop. The user-facing run needs a durable state such as blocked: invalid payload, a correlation to the original message and error, and an owner or repair route. Moving the queue message to a dead-letter queue is an operational step, not a completed business outcome.
First distinguish deterministic failure from a transient dependency fault. Validate schema and version at ingress when possible, and again at the worker boundary. For a malformed payload, capture a safe diagnostic and quarantine it after a small bounded attempt count. For a temporary database outage, use backoff, a deadline, and a retry budget. A broad catch that treats both as “try again” makes poison jobs indistinguishable from recoverable ones. Amazon SQS visibility-timeout documentation notes duplicate delivery under its at-least-once model, and its dead-letter queue guidance describes moving repeatedly failing messages aside. These mechanics do not decide the business state for us.
If the queue preserves order per account or workflow, moving one bad message aside may let later messages run in an invalid state. Do not blindly free the group. Decide whether later jobs depend on the missing transition. A subscription update following an unprocessed cancellation, for example, must be held or revalidated. Record a gap marker in the workflow ledger, alert the owner, and resume only when a corrected event or an explicit skip decision establishes the right state. For independent jobs, quarantine the bad one and allow others to proceed.
After a parser fix, redrive with the original logical job ID and source version. Check whether any earlier attempt performed a side effect before it failed. The worker may have sent an email, then crashed while parsing a follow-up result. Redriving that message as though nothing happened can duplicate the effect. Read the operation ledger or provider status first and replay only the safe portion. The queue receipt ID is not the business operation ID.
If a parser bug rejects a million valid jobs, the quarantine path has to handle volume without deleting evidence or overwhelming operators. Stop the bad worker deployment, preserve messages within retention bounds, increase capacity only after the parser is fixed, and redrive in controlled batches with monitoring. If retention may expire, move the durable payload and state to a protected store under the relevant policy. Do not set the receive count to infinity and call the backlog safe.
Useful measures are age of the oldest uncompleted user task, number blocked by deterministic errors, retry attempts per logical job, dead-letter age, and verified effects after redrive. Queue depth alone can fall to zero while thousands of users still have blocked runs. The release test should inject malformed payloads, a crash after an effect, and an ordered dependent job behind the poison one.
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →