Distributed Reliability · Staff
A million-row model batch fails at 70 percent. What exactly can you publish?
The question
Interview question
A nightly model job scores a million customer records. It writes some results, the provider batch expires, and the orchestrator retries the failed part. Meanwhile the source table gets corrected and the prompt changes. A downstream dashboard sees 700,000 rows and calls the run complete. How would you design the run and publication so retries cannot silently mix inputs, models or half-finished output?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
First I would separate an attempt from a dataset edition. A retry is a new execution attempt. The logical edition is defined by an immutable input manifest and a pinned configuration: source table version or exact row hashes, model and provider version where available, prompt and tool versions, decoding settings, output schema and policy revision. If any of those change, it is a new edition or an explicitly approved repair, not a transparent retry of the old one.
Each input record gets a stable logical ID within the edition. For every ID, track a terminal state such as succeeded, permanently failed, excluded with reason or pending. A provider's request ID and the local attempt ID are separate from that logical ID. Output order is not a join key. The OpenAI Batch API guide tells clients to use custom_id to map output lines back to input rather than relying on order, and it documents separate error results for requests that do not complete. The design applies to any provider, but the exact retry behavior and model version guarantees must be checked for that provider.
Write attempts to a staging location keyed by edition, logical ID and attempt. Validate schema and semantic requirements, then choose one winning terminal result per logical ID with a deterministic rule. A duplicate provider completion must not create a duplicate published row. If a call may have performed an external side effect, this scoring pattern alone is insufficient. The effect needs its own idempotency and reconciliation contract. Here the model is producing a score, so recomputation is possible, although it may cost money or yield a different answer.
The downstream dashboard reads only a committed manifest, never the staging table. The manifest declares expected count, eligible count, success count, excluded and failed IDs, content checksums, source and model provenance, and whether the edition meets its publication policy. Commit it atomically with a pointer swap or a transactional table update that the dashboard respects. If policy requires 100 percent coverage, 700,000 rows cannot be “complete.” If a partial release is useful, give it a distinct label, explicit coverage and per-record absence semantics so it cannot masquerade as a full nightly run.
The corrected source creates a real choice. We can finish the original edition against the pinned old source and label it as such, or cancel and restart a new edition with the correction. We should not fill the remaining 300,000 rows from the new table under the old edition name. Similarly, a prompt revision belongs to a new edition. A provider may not offer immutable model revision pinning, in which case record the most precise model identifier and run interval available and be honest that byte-identical replay is not guaranteed.
If the interviewer pushes on a crash exactly after the manifest commits but before the scheduler records success, I would make commit idempotent and have restart query the published manifest. If it exists with the same edition digest, report success without republishing. If the digest differs, stop and investigate. Watch completeness, duplicate IDs, schema rejection, cost per successful row and publication lag. The core guarantee is that a consumer sees one declared edition with known coverage, not a convenient-looking pile of rows.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →