Skip to content

Elasticsearch acknowledged the document. Why can't search find it?

Search Returned an Answer. Was It the Right One? · Part 3 of 6 · 5 min read

Read the worked explanation ↓

Interview question. A retailer indexes a new product. The index API returns success, a direct lookup by ID can find it, but a search for its exact product name returns nothing for a short period. An engineer says the write must have been lost and proposes forcing a disk flush after every product update. Explain what really happened and how to meet a five-second search-visibility target without destroying ingestion performance.

There are at least three different moments in this story. The system may have accepted and durably recorded an indexing operation. It may have made the new document visible to a searcher. And it may eventually have persisted a new Lucene commit point. Calling all three of those things "committed" causes needless incidents.

A search index is often built from immutable segments. Newly indexed data becomes searchable when a refresh opens an appropriate new search view. A successful index response by itself does not promise that the next search sees the new document. For Elasticsearch, real-time document GET and ordinary search have different visibility behavior. That difference explains how an ID lookup can succeed while a matching search temporarily misses the product.

An index write receives an acknowledgment through the translog path, then a refresh makes a new segment visible, while a flush and merges serve separate purposes.
An index write receives an acknowledgment through the translog path, then a refresh makes a new segment visible, while a flush and merges serve separate purposes.
Think first

A travel search service indexes 60,000 hotel rate changes per second, and users expect a new rate within three seconds of the authoritative update. Design ingestion measurements, refresh strategy, version checks, load testing, and fail-safe pricing validation. Explain the cost of per-write refresh.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

The write and the searcher follow different schedules

The primary shard handles the indexing request and coordinates replicas according to the engine's write protocol. Lucene builds the searchable representation, while the translog supports recovery from operations not yet included in a Lucene commit. Under Elasticsearch's default request-level translog durability, the system fsyncs the relevant translogs before acknowledging a successful request. Configuring asynchronous translog durability changes that guarantee and can expose acknowledged operations to loss after a failure.

A refresh is a different operation. It makes recently indexed documents visible to search without requiring a full Lucene commit. Background refresh timing depends on settings and whether shards are actively searched. Elastic's general documentation describes a one-second default in the Elastic Stack and different defaults for Serverless, with search-idle optimizations. Those are defaults, not a per-request deadline you should promise without measuring the entire pipeline.

A flush performs a Lucene commit and starts a new translog generation. Flush and refresh answer different questions. Forcing a flush to make one product searchable confuses recoverability with visibility. Flush is not the appropriate routine tool for low-latency search publication.

Segments are also not rewritten in place for every update. New segments and deletion markers accumulate. Background merges can combine smaller segments into larger ones, reclaiming some deleted data and simplifying future searches. Merging consumes disk IO, CPU and temporary space. It is not a prerequisite to making each new document searchable. If the team forces a refresh for every single write, it can create many tiny searchable segments and increase merge pressure later.

Walk the exact user-visible timeline

At 12:00:00.000, the product service receives an update for product P-482. It writes to the authoritative catalog and a separate indexing path sends version 18 to Elasticsearch. At 12:00:00.080, indexing receives an acknowledgment. A direct real-time GET may show version 18. At 12:00:00.250, a query runs against a searcher that still sees the previous refresh snapshot. It returns no hit. At 12:00:01.100, the affected shard refreshes and later searches can see version 18, assuming the query, analyzer, filters and routing also match.

Those times are illustrative, not an Elasticsearch service-level guarantee. In a real deployment the product update may also wait behind a queue, retry an indexer, or search a replica with a different current view. Measure each stage. If the application promises "searchable within five seconds," start the clock at the authoritative catalog commit, not at an internal indexing request that might have sat in a queue for four seconds already.

What would I do for a user who creates a listing and immediately opens its page? I may provide read-your-write behavior by serving the detail view from the catalog or a version-aware preview, not by forcing all search infrastructure to synchronously publish every document. If the workflow explicitly requires searching the new listing before responding, Elasticsearch supports refresh=wait_for on relevant write APIs. It waits for a refresh rather than forcing one immediately. It affects response latency and has operational caveats, so I'd reserve it for workflows with that contract.

Test the trade-off under actual update traffic

Forcing refresh=true after every request can make indexing appear correct in a trivial test while harming production throughput. I would track indexing latency, age of the oldest unapplied source change, refresh time, segment count, merge backlog, search p99, translog flush/replay behavior and version visibility from multiple search replicas. I'd inject sustained bulk indexing while search is live, then measure a percentile of end-to-end freshness and the worst-case tail.

Updates and deletions also deserve separate tests. A modified product might be visible with old fields for a short time. A deleted product might linger until refresh, which is a different issue from whether an operation is durable. Safety-sensitive removal or authorization revocation requires stronger protection, potentially at the query gateway or authoritative store. Do not claim that a one-second refresh interval provides an immediate access-revocation guarantee.

The interview question is really about naming the correct boundary. Acknowledged, searchable, durably committed to a Lucene point, and fully merged are not the same state. Once the team draws those as separate stages, the right fix becomes a measured freshness policy, not an expensive flush after every click.

Go deeper

Elasticsearch's indexing APIs can use refresh=true, refresh=wait_for, or the default behavior. An immediate refresh makes changes visible but may create inefficient tiny segments under heavy write rates. wait_for ties response completion to a future refresh without forcing one as a general rule. If the shard’s refresh-listener limit is exhausted, it can force a refresh; with automatic refresh disabled, it can wait indefinitely until something refreshes the shard. Translog durability settings change crash guarantees independently. Search can target multiple shard copies, so an end-to-end visibility objective needs representative probes and replica-aware measurement, not just one successful GET. A Lucene commit is not identical to a search refresh.

Check your reasoning

  • Separate acknowledged indexing from searcher visibility.
  • Explain translog durability and real-time GET versus search.
  • Distinguish refresh from Lucene flush and segment merge.
  • Trace source-commit-to-search freshness instead of assuming refresh_interval is a hard SLA.
  • Choose wait_for selectively and protect access revocations with stronger controls.

Common mistake: Flush after every write is a cheap way to guarantee search refresh.

An acknowledged document is available through real-time GET but absent from search. Which response addresses the correct visibility boundary?

Read the answer explanation

Budget source-to-search delay and selectively wait for refresh where required — Refreshing searchers, committing durable Lucene state, and segment merging are separate mechanisms.

Apply it elsewhere

An internal permissions catalog revokes an engineer's access to a document. The search index updates within two seconds. Design a read path that prevents unauthorized results during the two-second window and distinguish hiding a hit from protecting its downloadable content.

Compare your approach

Separate write acknowledgment, translog durability, searcher refresh, Lucene flush and segment merge, explain direct GET versus search visibility, budget end-to-end freshness from the source commit, and avoid equating refresh configuration with a hard revocation SLA.

One more constraint

Does forcing a Lucene flush after each acknowledged update provide the same mechanism as making the update visible through refresh?

Quick revision

Elasticsearch can acknowledge an indexing request before a refreshed searcher sees it. A real-time GET may see the document while search uses the previous view. Translog durability supports recovery, refresh publishes a new searchable view, flush records a Lucene commit, and background merges consolidate segments. Do not flush per write to solve visibility. Measure source-to-search freshness, choose normal refresh or targeted refresh=wait_for, and handle critical revocations or price checks against authoritative state.

References