The queue is doing its job: preserving work. The service may still be failing its promise. If a job has a ten-minute useful life and sits behind forty minutes of work, processing it later is wasted capacity, even if every worker and queue health check is green.

I want a quick estimate of work debt, not just message count. Measure incoming work per minute, sustainable completed work per minute at the reduced model capacity, backlog age and size by class, and the expected work remaining in the queue. For model jobs, a thousand short classifications and a thousand long generations are not the same debt. The estimate can use tokens and measured service time per class, then improve as completions arrive. Include retries and jobs already in flight so the apparent arrival rate is not understated.

Here is the simple version of the math. Suppose comparable jobs arrive at 120 per minute, the reduced fleet completes 90, and 3,600 are already ahead in a FIFO queue. New arrivals see about forty minutes of work ahead of them at 90 per minute. The debt grows by 30 jobs a minute while intake stays at 120. If we limit accepted new work to 70 per minute, the backlog shrinks at roughly 20 per minute. Clearing 3,600 jobs takes about 180 minutes, not forty. If accepted work stays at or above service capacity, the backlog never drains. These are estimates for comparable jobs and steady rates. With variable token lengths, make the same calculation in estimated service time and show an uncertainty band.

I would change intake as soon as this is visible. A new ten-minute job cannot be accepted into a forty-minute FIFO wait with a promise of on-time completion. Reject or defer it before durable acceptance, tell the caller the capacity condition and a realistic retry path, and rate-limit retries so clients do not turn backpressure into more queue work. If there is a small reserved lane for short deadline jobs, admit only the amount it can actually finish before their deadlines. An “urgent” flag with no available service budget just moves someone else's miss around.

Existing jobs need a decision based on their contract. Sweep the queued jobs for expired deadlines and terminally mark those whose result has no value after expiry. Notify the owner and preserve an audit record. Do not silently delete messages to make the queue graph look better. For a job that must complete even when late, keep it and communicate the revised time. For a job with a business deadline, estimate its start and finish against remaining capacity, not only the age of the oldest message. Running jobs with external effects need cancellation and reconciliation rules before we stop them. A queued inference that has never begun is much easier to expire safely.

The interviewer asks whether I would jump all ten-minute jobs ahead of the old queue. Not automatically. Some old jobs may have earlier deadlines or contractual priority. Earliest deadline first can help when jobs have known work and are feasible, but it cannot make an overloaded system feasible. I would use explicit service classes, their reserved capacity or priority policy, and admission that accounts for work already promised. Otherwise a stream of new urgent jobs can starve old work forever. We may deliberately shed stale, low-value work so fresh useful jobs can run, but that is a product decision with visible outcomes.

I would also check what capacity can safely be restored or moved: a compatible pool, a known smaller model for tasks where its quality contract permits substitution, or a temporary reduction in optional generation length. Do not silently send a high-stakes task to an unvalidated model or lower its output quality just to improve throughput. And don't add workers beyond the model server's real throughput. That can increase contention and make service slower.

During recovery, the dashboard should show arrival work rate versus completed work rate, estimated time to drain at the current net rate, age and deadline misses by class, expired jobs actually executed, in-flight retries, and admission rejects. AWS's queue guidance calls out message age and stale work as signals. An oldest-message metric alone still cannot tell a caller whether their job will meet its own deadline. I would replay the capacity drop and gradual restoration with a mix of short and long jobs, then verify that expired work stops consuming scarce model capacity and that new admission promises match observed completion times.