Skip to content

The new index has two billion documents. Can we switch search to it now?

Search Returned an Answer. Was It the Right One? · Part 6 of 6 · 6 min read

Read the worked explanation ↓

Interview question. A marketplace needs to change its product analyzer and field mappings. The live search index holds two billion documents and receives thousands of updates per second. The reindexing job finished copying the historical data into a new index. Engineers propose atomically swapping the read alias and deleting the old index. Explain why the copy can be complete yet the new index still be wrong, and design a safe migration.

I would ask one question before approving the swap: Which product version would we serve for an item updated during the copy? A bulk scan does not freeze a changing source of truth just because the copy reaches its last row. A document read at 09:00 may have been updated at 09:05, deleted at 09:06, and re-created at 09:07. If the new index reflects only the original snapshot, changing an alias just makes an old world appear instantly.

The alias switch is the easy part. Keeping writes, deletions and ordering correct on both sides of the switch is the migration.

A versioned source snapshot and change stream converge into a new index before an atomic read-alias cutover, with ownership and rollback conditions checked separately.
A versioned source snapshot and change stream converge into a new index before an atomic read-alias cutover, with ownership and rollback conditions checked separately.
Think first

Rebuild a two-billion-document employee directory under a new multilingual analyzer while ten thousand document changes per second continue. Design snapshot watermarking, CDC replay, version and deletion rules, read and write alias ownership, consistency validation, cutover and rollback.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Build a new representation without changing the old one

First create a separate destination index with the new mapping, analyzers, shard arrangement, lifecycle settings and capacity. Reindexing data does not automatically copy all of those settings. Analyzer changes typically require rebuilding stored tokens.

Choose an authoritative data source. If product data comes from a transactional catalog database, prefer a consistent snapshot of that catalog plus an ordered change feed. If Elasticsearch is the only copy, account for its version and deletion semantics. A catalog plus CDC provides a clearer durable source sequence.

At snapshot time, record a source high-watermark, say position L0, using a method that couples snapshot consistency and change capture without gaps. Copy every current product as of that snapshot. At the same time, preserve later changes beyond L0 for replay into the destination. The snapshot may take hours. Retain change history long enough and reject stale writes by document version.

The deletion that a naive reindex misses

Suppose the snapshot contains product 88 at version 40. During copying, a customer removes the listing at version 41. If the new index later receives the old snapshot document but never receives the delete, the deleted listing returns after cutover. The change history is incomplete.

Even if we capture the delete, ordering matters. If version 41 is applied before a slow snapshot worker writes version 40, an unguarded overwrite can resurrect the product. Use a version-aware indexing protocol appropriate to the source and destination, and keep sufficient deletion markers to reject late old writes. With out-of-order events across products, a global ordering may be unnecessary, but the sequence for each product must preserve or enforce its latest authoritative version. Do not claim Elasticsearch external versioning alone solves retention of all tombstones forever. Define the late-write window and the source recovery mechanism.

Another trap is treating two independent writes as a transaction. Dual-writing every live change to both old and new indices introduces partial successes, timeouts and differences in refresh timing. Dual writes need durable retry and reconciliation. Replayable change streams are easier to reason about.

Cutover is a protocol, not an API call

When the destination snapshot is complete and its replay stream is caught up, measure the lag from source commit to destination visibility. Check counts by partition or tenant, sample document versions, missing deletes, analyzer behavior and relevance quality. Counts alone are weak because an extra stale item and a missing current item can cancel numerically. Perform source-versus-index reconciliation on identities and versions.

Now prepare the read cutover. Elasticsearch aliases support changing index targets through the aliases API. A single atomic alias update can move the application's read name from the old index to the new one. But the API does not automatically route independent application writers to the correct index or drain CDC work. If writers use a different write alias or a hard-coded physical index, coordinate those paths separately. Account for in-flight requests.

A safe plan might keep indexing both versions for a limited compatibility window while only the read alias moves. Verify old-index writers cannot silently become authoritative. If the new index fails after cutover, rollback means more than pointing reads back. The old index must still have complete updates since the switch, or we must replay them into it before using it again. Otherwise rollback swaps one incomplete dataset for another.

Test the migration under concurrent creates, updates, deletes and updates to documents that share analyzers. Induce an indexer restart between checkpoint and acknowledgment. Verify retries converge on the correct product version and that the source log can recover after a prolonged pause. Check source retention before beginning a multi-day copy, not after it exhausts history.

Decide what counts as success

The new index can be functionally different by design. Its analyzer may change match results, score distributions and tokenization in ways that are not detectable by document count. Shadow a representative set of user queries against both versions, compare eligible result sets and relevance metrics, and have owners approve intentional changes. Privacy filters and deletions deserve stricter tests than ranking. A product that became restricted must not reappear simply because the migration got its row count right.

I'd also keep the old index for a bounded, costed fallback period with explicit criteria for removal. Monitor index refresh lag, source-to-destination version lag, stale-result rates, error budget, alias targets and unexpected writes to the retiring index. Rollback depends on recoverable history.

The interview lesson is atomic routing does not imply atomic data migration. A trustworthy reindex couples a consistent snapshot with complete change capture, ordered or version-safe application, validation of deletions and an operationally sound cutover. If those conditions hold, the alias switch becomes the uninteresting part of the design.

Go deeper

Elasticsearch's _reindex API copies documents from a source to a distinct destination and requires the target mappings and settings to be defined separately. It is not a full application CDC protocol. _aliases actions can atomically change alias targets, and is_write_index determines write routing when an alias points to multiple indices. A PIT or reindex source snapshot alone does not preserve all ongoing changes until cutover. External versioning, sequence ordering, tombstones, retry behavior and source log retention need explicit design and tests. Rollback requires an up-to-date former target or replay into it.

Check your reasoning

  • Establish consistent snapshot and matched CDC watermark.
  • Apply inserts, updates and deletes with monotonic per-ID versions.
  • Reconcile identities and versions, not just total document count.
  • Separate atomic read alias switch from write-index ownership and indexer checkpoints.
  • Prove old-index catch-up and rollback while change history is retained.

Common mistake: A completed historical reindex plus an atomic read alias switch guarantees no lost updates.

A historical reindex completed while live updates continued. Which sequence can establish a safe cutover?

Read the answer explanation

Backfill, replay live changes, validate versions and deletions, then coordinate alias and writer cutover — An atomic routing change does not carry source updates made during a long-running copy. Replay and migration validation do.

Apply it elsewhere

Migrate an authorization decision cache between two storage engines without exposing revoked permissions. Adapt the snapshot-and-log protocol and explain which authorization checks must stay on the authoritative path during migration.

Compare your approach

Explicitly handle updates/deletes after snapshot, prevent stale snapshot overwrite, distinguish CDC checkpoints from visibility, separate read and write cutover, require content/version reconciliation, and explain why rollback needs continued catch-up.

One more constraint

If an atomic read alias swap succeeds, does it prove the destination index contains every update and delete committed during the historical copy?

Quick revision

A completed snapshot copy can be stale before its final row is written. Build the new search index with new mappings, capture a consistent source watermark, backfill the snapshot, replay every later change including deletes, and enforce per-document version monotonicity. Validate source-versus-index identities and versions, query relevance and privacy, and destination lag. Move the read alias atomically only after the destination converges, coordinate writers separately, and retain enough history to make rollback genuinely safe.

References