I would take the result seriously, but I would ask what the recalled unit means. A sentence can match the query exactly and still be unusable evidence. “The period is 30 days” may depend on a heading, a table column, and the preceding line saying “Except for enterprise contracts.” If we hand that sentence alone to the answerer, recall improves on paper while the answer becomes less grounded.

There are two different tasks hidden in “chunking.” One is finding a small relevant span. The other is giving the model enough coherent, attributable context to answer. Sentence vectors can be useful for the first. They need not be the unit we cite or send to the model. I would compare sentence retrieval followed by expansion to its parent paragraph or section against paragraph retrieval with better exact search and reranking. Keep page, section, table, and source-version identity through either path.

The cost can change by a large factor. Imagine a corpus with twenty useful paragraphs per document and about a hundred sentences. That is roughly five times as many embedding records before any overlap or OCR duplication. It means more embedding work on updates, more index metadata and maintenance, larger candidate sets, and more chances for several nearly identical sentences to crowd out another source. The actual bytes and latency depend on the index and compression, so I would measure them, not present the fivefold record count as a fivefold hardware bill.

The evaluation has to count the evidence the answer needs. Does the correct clause and its exception reach the final six passages? Are headings and units preserved? Can the answer cite a span that actually supports each claim? Measure retrieval at a fixed latency and candidate budget, then answer support and wrong-answer rate. Slice by exact identifiers, multi-sentence questions, tables, scanned PDFs, and document versions. A small test set that asks for facts already contained in one sentence is biased toward sentence chunks.

I would inspect what the old paragraph splitter did badly before changing the entire corpus. It might cut through a table, join unrelated headings, or use paragraphs so long that a specific error code is diluted. Fixing that parser or adding a lexical branch may buy the same recall without a billion tiny vectors. Google's Document AI layout parser documentation is one concrete example of treating headings, tables, and document layout as retrieval structure instead of reducing everything to raw line breaks.

Now OCR doubles the number of “sentences.” I would look at a sample of the new records. Are they actual text, broken words, headers repeated on every page, or table cells split into fragments? Embedding Invo1ce shal1 b3 pa1d as though it were a clean sentence can waste capacity and create noisy matches. Decide which pages need OCR, keep confidence or extraction-quality signals where available, and quarantine obviously broken output for a better extraction path. Do not quietly drop a scanned page that contains the only governing exception. Mark it as incomplete evidence and route it for repair.

Suppose the sentence approach wins strongly on rare error-code queries. I might use a targeted sentence or line-level index for that slice, especially with lexical matching, while keeping larger parent units for answer construction. The index does not have to use one granularity for every document type. An exact stack trace, a policy table, and a narrative design document do not behave the same way.

I would approve a staged experiment, not a global rebuild on the first recall chart. Keep the old index for rollback, evaluate end-to-end with stable source snapshots, and inspect the answer and cost slices together. The choice is about how much usable evidence reaches the model for the cost we can afford, not which chunk size wins one ranking metric.