A customer updates an internal security policy at 10:02 in the morning. The source system confirms the change and the new version enters the ingestion pipeline. At 10:07, an employee asks a question that depends on the updated policy. The RAG system retrieves chunks from the previous version and gives an answer that is now wrong.

At the same time, another tenant starts importing two million documents. The embedding queue grows, normal updates wait behind the bulk job, and deletion events slow down as well. Some documents now have chunks from two versions because an indexing worker retried after a timeout. The source database says one thing, the search index says another, and nobody can tell which version was used in the answer.

This is a realistic failure shape for a large RAG platform.

With 50 million source documents, the system may be storing 300 million, 500 million, or more chunks. Documents are added, changed, deleted, reprocessed, and moved between permission groups. Embedding models change. Chunking logic changes. Search indexes are rebuilt. Tenants have very different corpus sizes and traffic patterns.

The design needs to preserve retrieval quality while all of this is happening.

Start with the indexed corpus, not the document count#

Retrieval runs over chunks.

Assume 50 million documents and eight chunks per document. That gives us 400 million searchable chunks. With 1024 dimension embeddings, a float32 vector needs roughly 4 KB and a float16 vector needs roughly 2 KB. The raw vector data is therefore around 1.6 TB with float32 or 800 GB with float16.

That number does not include the search index, graph structure, metadata, lexical index, replicas, stale records waiting for cleanup, or the second index generation used during a migration. The real storage footprint can be several times larger.

Write traffic also arrives in bursts. If one percent of the corpus changes each day, the platform receives around 500,000 document updates and roughly four million chunk updates. The average rate may look reasonable, but connectors run scheduled scans, customers perform bulk imports, and failed jobs are replayed. Capacity planning needs to cover both normal traffic and recovery traffic.

A full model migration gives a clearer view of the background load. Rebuilding 400 million embeddings in seven days requires about 661 successful embeddings every second. A thirty day migration still requires about 154 every second before retries and new documents are included.

Ingestion, serving, and migration traffic should not compete for the same uncontrolled pool of capacity.

End to end architecture of a multi tenant RAG system showing versioned document ingestion, vector and lexical indexing, tenant aware routing, hybrid retrieval, permission validation, reranking, and answer generation.
End to end architecture for a multi tenant RAG platform with versioned ingestion, hybrid retrieval, freshness controls, and background reconciliation.

Use a document registry as the source of truth#

The vector index should not be the authoritative record of the corpus. It is built for retrieval, not for coordinating document versions, permissions, deletes, processing state, and model migrations.

The system needs a document registry.

For each document, the registry stores the tenant, source identifier, document identifier, source version, content hash, active version, desired version, permissions, parser version, chunker version, embedding model version, index generation, and processing state.

The original file or a normalized immutable copy lives in object storage. The registry points to that object and records how it was processed. The vector and lexical indexes contain derived data that can be rebuilt.

This gives the platform a clear answer to three operational questions.

What should exist?

What currently exists?

Which version is active for retrieval?

Without this registry, teams end up inferring state from queues, index records, and logs. That works until a retry creates duplicate chunks or a partial failure leaves half a document indexed.

Build ingestion as a versioned state machine#

Every source change should become a durable event with a stable identity. An upsert event includes the tenant, document identifier, source version, source timestamp, content location, and permission information. A delete event includes the same identity and the version that caused the deletion.

The pipeline moves through source capture, durable event log, document fetch, parsing, normalization, chunking, embedding, index writing, validation, and activation.

Each stage records progress against the same immutable document version. The idempotency key can be derived from the tenant, document identifier, source version, and processing configuration. When a worker retries, it writes the same logical output instead of creating another active copy.

This matters because the parser, embedding provider, registry, object store, lexical index, and vector index do not participate in one shared transaction. Even when the event platform provides strong delivery guarantees, the complete pipeline still needs idempotent writes and reconciliation.

Each stage should persist reusable output. If embedding fails, the system should reuse the normalized chunks instead of parsing the file again. If vector indexing succeeds but lexical indexing fails, the document remains pending until both are complete.

Malformed files need a quarantine path with the failure reason, parser version, retry count, and last successful stage. Sending the same corrupt file through the main queue again and again only consumes capacity and hides useful work behind it.

Schedule ingestion by workload and tenant#

A single global queue allows a bulk importer to control freshness for every other tenant.

Use separate scheduling classes for deletes, normal updates, new document ingestion, tenant backfills, and platform migrations. Deletes receive the fastest path because removed content remaining searchable can become a security issue. Normal updates keep reserved capacity so that they continue to move during backfills. Migration work uses controlled background capacity.

Within each class, scheduling should be tenant aware. A weighted fair scheduler gives every active tenant progress while allowing larger customers to receive higher reserved capacity.

Limits should cover documents per second, bytes per second, chunks per second, embedding tokens, concurrent parser work, and outstanding queue depth. Admission control belongs before a very large import enters shared processing.

Queue age is usually more useful than queue length. Ten thousand small text files and ten thousand scanned PDFs represent very different workloads. Track the age of the oldest item, bytes waiting, estimated tokens waiting, and expected compute time at every stage.

Prepare a complete document version before activation#

A chunk record should be immutable and traceable to the document state that produced it. Its identity should include the tenant, document identifier, document version, chunker version, and chunk position or content fingerprint. It also carries the embedding model version, permissions, checksum, and index generation.

The pipeline creates a manifest for every document version. The manifest lists the expected chunk identities and count. New chunks are written as pending. Activation happens only after the vector and lexical indexes both contain the complete manifest.

Once validation succeeds, the registry changes the active document version in one transaction.

The search indexes may temporarily contain old and new versions. The retrieval service therefore checks candidates against a replicated cache of active versions before reranking and context assembly. Old chunks are removed after activation.

This design avoids requiring an atomic update across many search records. It also gives the platform a clean rollback. If a parser release produces bad chunks, the registry can move the active pointer back to the earlier version while cleanup happens in the background.

The retrieval service should request more candidates than it finally needs because some candidates may belong to inactive versions. The overfetch amount should be based on the observed stale record ratio.

Document version lifecycle showing a new version written to vector and lexical indexes, validated against a chunk manifest, activated atomically, and later cleaned up, with a separate path for logical deletion.
A new document version remains pending until both indexes contain a complete manifest. Deletes become invisible to retrieval before physical cleanup finishes.

Make deletes invisible before physical cleanup#

A deletion should stop appearing in queries quickly, even when removing every physical copy takes longer.

When a delete event arrives, the registry marks the document deleted and publishes the identity to a low latency tombstone cache used by the query path. Retrieval results are checked against that state before any content reaches the model.

Removal from the vector index, lexical index, object storage, caches, and replicas continues afterward. The registry remains authoritative and the tombstone cache must be recoverable from durable state so that a cache loss cannot make deleted content visible again.

Measure source deletion to registry tombstone, registry tombstone to query invisibility, and query invisibility to physical removal. These are different stages with different operational meaning.

Partition around tenant routing and bounded fanout#

At hundreds of millions of chunks, the search layer will be partitioned. The query router needs to know exactly which partitions to contact.

A pure document hash distributes data evenly, but tenant queries may require broad fanout. A pure tenant partition simplifies routing, but a few large tenants can create oversized partitions and hot nodes.

A practical design uses tenant aware virtual partitions.

Small tenants share physical partitions. Large tenants receive several virtual partitions, with their documents distributed by document hash. A routing catalog maps each tenant to the correct vector and lexical partitions.

This allows the platform to move a large tenant, add partitions, or place a regulated tenant on dedicated infrastructure without changing document identities. It also avoids a separate physical index for every small tenant.

If one tenant spans many partitions, add a second routing dimension such as collection, product area, region, language, or time range. These routing filters need evaluation because a missed partition cannot be repaired by reranking.

Replicas should be spread across independent failure domains. The router needs health aware replica selection and a strict fanout budget.

Tenant aware partitioning design where small tenants share search partitions, large tenants span multiple virtual partitions, and the routing catalog limits queries to relevant vector and lexical shards.
Small tenants share partitions, large tenants span virtual partitions, and the routing catalog limits each query to the relevant search targets.

Choose the vector index from measured workload#

The correct vector index depends on corpus size, memory budget, update rate, recall target, and latency target.

HNSW often provides strong approximate nearest neighbor performance, but the graph and vectors can require substantial memory at this scale. Disk based designs can reduce the memory footprint by keeping more of the index on SSD, though that changes the latency and operational profile.

Compression also needs measurement. Float16, scalar quantization, and product quantization can reduce storage and memory, but the effect on recall depends on the embedding distribution and the retrieval task.

Compare index choices using recall, p95 latency, p99 latency, update throughput, rebuild time, memory use, failure recovery time, and cost per thousand queries. The useful question is whether relevant evidence still reaches the reranker within the latency budget.

Run lexical and vector retrieval together#

A query starts with authenticated tenant and user context. The retrieval service resolves the tenant, permission groups, allowed collections, active index generation, and routing targets before it sends any search request.

Lexical and vector retrieval then run in parallel.

Lexical search remains useful for identifiers, error codes, product names, exact phrases, policy numbers, and rare terms. Vector search handles semantic similarity and paraphrased questions. Their rankings can be combined using reciprocal rank fusion or another rank based fusion method.

The query path becomes authentication, query normalization, query embedding, parallel retrieval, rank fusion, active version validation, permission validation, reranking, context selection, generation, and citation assembly.

The first stage should retrieve enough candidates for the reranker without sending hundreds of weak results through an expensive model. The candidate count should come from recall and latency measurements.

Context assembly should work at document level as well as chunk level. Several adjacent chunks from one document can crowd out evidence from other sources. The assembler can group related chunks, remove duplicates, expand nearby context when required, and enforce limits per document.

Apply authorization during retrieval#

Tenant isolation cannot depend on a prompt instruction or a filter supplied by the application caller.

The retrieval service should inject tenant and permission filters from authenticated server side context. These filters should run inside lexical and vector search where possible, so unauthorized records do not enter the candidate set.

Filtering only after nearest neighbor search can reduce recall because the search may return many candidates that are later rejected. The index and query strategy should support filtered retrieval that still returns enough authorized results.

Before context reaches the model, every selected chunk should be validated again against tenant, permission, active version, and delete state. This second check protects against stale permissions, routing mistakes, index corruption, and software defects.

Every cache key should include tenant identity, permission scope, active corpus generation, embedding model generation, and query normalization version. A shared semantic cache without this scope can become a cross tenant data path.

Carry one deadline through the query path#

End to end latency is usually controlled by the slowest parts of the search fanout.

The query coordinator should carry one deadline and assign budgets to query embedding, lexical search, vector search, reranking, context loading, and generation. Each downstream service receives the remaining deadline instead of using an unrelated timeout.

The coordinator also needs an explicit degradation policy. If the reranker is unavailable, the system may use fused retrieval rankings and lower the confidence of the answer. If one replica is slow, the router can try another within the remaining budget. If some partitions are unavailable, the response should record incomplete retrieval rather than presenting the answer as fully grounded.

Hedging can reduce tail latency for idempotent search requests, but it should begin only after a measured delay and count against the tenant query budget. Uncontrolled hedging increases load during an incident and can make a slow partition worse.

Track fanout, slowest partition time, candidate counts before and after filtering, reranker queue time, context bytes, model tokens, and candidates removed by version or permission checks.

Define freshness as an observable contract#

Near real time is not a useful production SLO.

For every change, record when the source changed, when the platform detected it, when the new version became active, and when the old version was physically removed.

These timestamps produce source detection lag, processing lag, activation lag, delete visibility lag, and cleanup lag. Each should have p50, p95, and p99 targets by workload class.

A normal update may have a five minute activation target. A delete may require query invisibility within seconds. A ten million document backfill may have a completion target measured in hours or days. These are separate contracts and should appear separately in dashboards and alerts.

The query trace should also record the corpus generation and source timestamps for cited documents. That makes stale answer investigations much easier.

Migrate models through separate index generations#

Overwriting embeddings in place makes rollback and diagnosis difficult.

For an embedding migration, create a new index generation with its own model version, chunking version, distance configuration, and build state. New documents can be written to both the serving generation and the new generation while background workers process the older corpus at a controlled rate.

Run production queries against both generations. Compare judged retrieval recall, overlap of top results, reranker outcomes, answer support, latency, and cost. A generation becomes eligible for serving only after it reaches the required corpus coverage and passes evaluation.

The router then moves tenants gradually. Start with internal traffic, then selected tenants, then a broader percentage. Keep the old generation available until the new one has been stable long enough for rollback risk to be acceptable.

Old and new embeddings should remain in separate generations unless compatibility has been proven. A model version changes the meaning of the vector space.

Embedding model migration using separate index generations, dual writes, background reembedding, shadow evaluation, gradual traffic rollout, and rollback to the previous serving generation.
A new embedding generation is built and evaluated beside the serving generation before traffic moves gradually with rollback available.

Reconcile desired state with indexed state#

Retries make ingestion recoverable. Reconciliation proves that recovery completed.

A background reconciler compares the document registry and manifests with both search indexes. It finds missing chunks, duplicate chunks, incomplete active versions, deleted documents still indexed, permission mismatches, and records using the wrong generation.

Safe discrepancies can be repaired automatically. Ambiguous cases should be quarantined.

This matters after incidents. A queue may report that every event was consumed while an index rejected writes during part of the outage. Consumer offsets do not prove that the corpus is correct.

Periodic retrieval checks add another layer. Select active documents, construct queries from known passages, and verify that the expected document is retrievable under the correct tenant and permission scope. Then verify that another tenant or an unauthorized user cannot retrieve it.

Isolate capacity across the whole system#

A tenant consumes connector capacity, parser compute, embedding tokens, index write throughput, vector memory, lexical storage, query fanout, reranking compute, model tokens, caches, and operational attention.

For ingestion, use tenant queue limits, weighted scheduling, and separate backfill capacity. For retrieval, use query rate limits, concurrent query limits, maximum fanout, candidate limits, and reranking budgets. For generation, reserve tokens using estimated prompt and output size. For storage, track document count, chunk count, vector bytes, index overhead, replicas, and stale generation bytes.

Large tenants may require dedicated partitions or clusters. That should remain an operational placement choice behind the same logical API.

The control plane manages tenant configuration, routing, quotas, connector state, model generations, and placement. The data plane performs ingestion and retrieval using cached control state. Query serving should not require a synchronous control plane lookup for every request.

Observe corpus correctness as closely as latency#

For ingestion, track source detection lag, queue age by stage, parser failures, embedding retries, index write failures, activation lag, manifest mismatches, duplicate chunks, stale version ratio, delete visibility lag, and cleanup backlog.

For retrieval, track candidate counts, filtered candidate counts, active version rejection rate, permission rejection rate, partition fanout, slowest partition latency, reranker latency, context diversity, citation coverage, and queries served with incomplete partitions.

For quality, maintain a judged retrieval set by tenant or domain. Measure recall before filtering, recall after filtering, reranker quality, answer support, citation correctness, and abstention behavior.

Every answer should be traceable to the query plan, index generation, embedding model version, retrieved chunk identities, active document versions, permission decision, reranker version, and model request. Without that trace, a wrong answer becomes an argument between the retrieval team and the model team.

The system I would build#

The final design has immutable source copies in object storage, a document registry for desired and active state, a durable event log, versioned processing stages, tenant aware schedulers, deterministic chunk identities, lexical and vector indexes, a routing catalog, an active version cache, a tombstone cache, a reconciliation service, and a query coordinator with deadline propagation.

The ingestion path prepares a complete document version before activation. The query path applies tenant and permission scope during search, validates active versions before reranking, and validates permissions again before generation. Deletes become logically invisible first and are removed physically afterward. Model migrations use separate generations with gradual traffic movement and rollback.

At 50 million documents, this structure gives the team a corpus whose state can be explained.

When a user receives a stale answer, the team can determine when the source changed, when the event was detected, which document version became active, which index generation served the query, which chunks were retrieved, and why they passed the permission checks.

That is the standard I would expect from a production RAG platform.