I would find a few failed queries with a known supporting page, then trace those PDFs alongside a recently successful PDF and a non-PDF control. The fact that latency stayed flat mostly tells me we did not obviously add a slow path. It does not tell me the right text was extracted, embedded, or allowed to reach the answerer. A fast search over bad content will still be fast.

Start with identity. Is the failed PDF actually in the corpus and the right tenant, version, language, and collection? Is it pending activation, quarantined, or routed to a different index generation? Can I open the exact source version cited by the ingestion record? If a new upload silently replaced an old file ID, I want to know whether the registry points to the old complete version or a half-built new one.

Then look at the artifact after each transformation, not only the job status. Count source pages, pages with extracted characters, pages needing OCR, text density, language, tables, and chunks per page. Open the page that contains the answer and compare the actual PDF with extracted text. An extractor can complete successfully while outputting a few headings, repeated footers, or a two-column page in the wrong reading order. A table value under the wrong header can produce a confident but false answer later. Apache Tika's PDF parser configuration exposes OCR strategy and page limits as examples of settings that can change what text is available, independent of whether the parse job exits successfully.

Next follow the known supporting span. Did chunking separate a qualifier from its rule? Was the page number retained? Did the embedding job receive that chunk or skip it because it was empty, too long, or marked duplicate? Were lexical and vector writes accepted, and did the index refresh? Can an exact source ID or phrase query find the chunk before ranking? If it exists in the index, inspect query filters, shard routing, initial candidate rank, reranker rank, context selection, and final answer. That lets me say where the evidence disappeared instead of arguing whether the embedding model is “bad.”

I would use a differential test around the introduction point. Reprocess an old good PDF through the new pipeline and a new bad PDF through the previous parser, if the old version is still available. Hold source bytes and permissions constant. If the new pipeline damages both, the parser or chunker change is a strong suspect. If only new PDFs fail under both, inspect their producer, scanned pages, fonts, encryption, or document layout. A new source may be delivering image-only pages, and an OCR configuration that worked for text PDFs will not rescue them.

But extraction reported success. What did the job contract call a success? No exception, one nonempty page, an expected page count, or evidence that a governing clause survived with its table relationship? Successful execution is not successful evidence. I would add quality checks that flag zero-text pages, abnormal chunk counts, missing page anchors, and large shifts in extracted language or character distribution. A threshold can flag a document for review, but it cannot prove the text is correct. Sampled visual comparison and query-based checks are still needed.

The corrective action depends on the stage. If parsing broke, stop activating the new generation, restore the previous parser for compatible PDFs, and reprocess the affected source versions. If the index has the right chunk but ranking loses it, test ranking changes only on the affected slice. If the source is unreadable, do not generate a confident answer from a nearby stale document. Keep the last valid version only if the product's freshness and citation policy permits it, and show that it is old.

Before reopening rollout, I would run a small PDF suite with text pages, scans, two columns, tables, footnotes, rotated pages, and a changed source version. For each, assert that the required span is extracted with its page and section, indexed, retrievable under the right principal, and used to support the final claim. The repair is done when those paths work, not when the extraction dashboard goes green again.