Forty percent of events does not tell us forty percent of work. If those edits mostly hit the same files, an event may replace another pending event. If they trigger OCR, embedding, and a full index rewrite every time, the tenant could consume much more than forty percent of capacity. I would locate the shared stage that is making the other tenants late before moving shards around.

Freshness needs two clocks. When did the source change become observable to us, and when did a complete version become searchable? Connector polling lag is different from a parser queue or an index refresh delay. I would break the age of each version down into detection, fetch, extract, embed, index, and activation, with queue age and actual work by tenant at each stage. A single global average hides the small tenants whose updates never get a turn.

For the repeated edits, I would keep one pending content job per tenant and source ID when the product only needs the latest current version. Deduplicate repeats of the same revision, and replace an unstarted revision 18 job when revision 19 arrives. There is little value in embedding 18 then. That does not mean discarding every event. A delete, a permission revocation, or a required audit transition has a different contract and must not wait behind a content compaction trick. Work already started needs a version check before activation so an old job cannot publish after the new one.

Then put a tenant-aware scheduler in front of the expensive stages. Limit in-flight fetches, OCR CPU, embedding tokens, and index writes separately. Give the smaller tenants a protected share, let idle shares be borrowed, and reclaim borrowed capacity when their own work arrives. A single round-robin over messages is not fair if one message OCRs a 900-page PDF and another parses a short Markdown file. The share has to be expressed in measured or estimated work, with estimates reconciled as jobs finish. SQS fair queues are one example of tenant-aware dequeueing, but the downstream OCR and model services still need their own capacity controls.

Partitioning solves a different part. Ten million documents should not force one tenant into one giant physical shard. Use stable document IDs and virtual partitions within that tenant, hashed or split along a queryable collection boundary. The routing catalog says which lexical and vector partitions hold the collection. This spreads writes and allows placement changes without changing source identities. I would check query fan-out as well, because a partition scheme that fixes ingestion but makes every search touch fifty shards just moves the pain.

I would keep backfills apart from live updates. A failed full scan of the ten million documents should not consume the service share reserved for newly changed files. If a source offers change tokens or versions, resume from a checkpoint and reconcile with a snapshot. The search path serves the last complete active version until the next one is ready. It should not cite half of a revised document just because the first chunks reached the index.

Now the tenant pays for a stronger freshness guarantee. That is a capacity promise, not a priority bit that skips everyone else's queue. Define exactly whether it covers source detection, searchable text, or both, and what document sizes and change rates the price includes. Reserve ingestion and index capacity for that class, or give the tenant an isolated pool if the burst and operational cost justify it. Charge for the actual OCR, embedding, storage, and rewrite work too. If source polling itself takes an hour, buying faster GPU service cannot produce a five-minute end-to-end guarantee.

Now a bulk import arrives while the paid tenant and the small tenants both have live edits. I would enforce the promised shares first, spend spare capacity on the import, and expose backlog and projected freshness per class. If even the reserved shares cannot meet their targets, stop accepting a stronger promise and say which stage is saturated. Hidden preemption of the small tenants would make the original problem look fixed on the paying tenant's dashboard while the platform stays broken.

The release test is a replay with repeated revisions, a ten-million-document backfill, urgent deletes, and a burst from several small tenants. Look at per-tenant source-to-searchable age, stale work discarded, queue work by stage, query fan-out, and cost. The scheduler is successful when a large customer's legitimate workload no longer makes another customer's freshness an accident of arrival order.