Understand what BM25 does, and what it does not
BM25 is a lexical relevance formula. Roughly, a matching term contributes more when it is rare across documents, more when it appears several times within the document with diminishing returns, and less when its frequency is explained by a very long field. Its common implementation uses term frequency saturation, inverse document frequency and document length normalization. That is useful evidence about text matching, not proof the product solves the customer's task.
A rare word like waterproof may contribute more than a ubiquitous word like boots, depending on the corpus. A document can rank highly because it repeats the right terms while belonging to the wrong category. BM25 does not inherently understand the difference between boots and a spray for boots. Query structure, field analysis, category filters, product metadata, and sometimes semantic or learned ranking provide that distinction.
I would inspect the top twenty results with an explanation of each scoring component. Are title matches overweighted, are descriptions filled with repeated keywords, and are brands unintentionally dominating? Check tokenization, plural handling and synonym expansion. Broad synonyms are especially dangerous. Treating "hiking boot" and "shoe accessory" as equivalent does not help relevance just because those categories are related.
Build a test collection before tuning knobs
Select a few hundred real queries with coverage across head, medium and tail traffic, new products, languages, misspellings, and exact product codes. For each, have qualified reviewers judge candidate results on a stable scale such as 0 for irrelevant, 1 for loosely related, 2 for useful, and 3 for clearly satisfying the intent. Record why a result received its grade. A boot accessory may be related but should not receive the same grade as a waterproof hiking boot.
Now evaluate ranking quality offline. Precision@10 asks how many of the first ten are relevant. NDCG@10 rewards placing highly graded results earlier and compares the achieved ranking with an ideal ordering. Recall@K measures whether useful results enter the candidate set at all. That last metric is critical. If the retrieval stage never considers a relevant boot, no re-ranker can rescue it. Don't confuse a perfect re-ranking model with good end-to-end search when retrieval misses the answer.
Suppose a query has ten useful results in your judged pool, and candidate retrieval brings back only six of them in its first hundred. Even an ideal re-ranker of that hundred cannot produce all ten. That is a recall limitation, not a scoring-weight problem. Increasing candidate breadth may improve quality while increasing latency, memory and downstream ranking expense. That's a trade-off worth measuring.
Offline judgments have biases. Popular queries dominate if you sample purely by traffic. Human reviewers disagree, and a result may be useful for one shopper but not another. Keep separate evaluation slices for categories, languages and queries with strict constraints. Measure changes against a frozen baseline before deploying.
Separate eligibility from preference
Waterproofing might be a soft preference for some queries and a strict product attribute for others. If the customer selects the "Waterproof" filter, the ranker must not override that eligibility decision with a high text score. Availability, tenant visibility, moderation and regulatory restrictions may also be hard filters. An attractive score must not smuggle in an item the user is not allowed to buy or see.
After eligible candidate retrieval, combine signals deliberately. Exact phrase in a title, category match, verified attributes, lexical score, stock status and limited business signals can affect order. Use learning-to-rank or a second-stage semantic model where the gain justifies its cost. Watch for feedback loops. Ranking popular items higher increases clicks, and future click-trained models may then mistake exposure for intrinsic relevance. Counterfactual or randomized evaluation can help, but it must respect the product's risk and user expectations.
Online A/B tests come after offline safety checks. Track successful product views or purchases, query reformulation, zero-result rates, abandonment, and latency. Don't optimize raw click-through in isolation. Misleading thumbnails and irrelevant but sensational results can win clicks and lose trust. Inspect regressions in smaller but valuable query segments.
The answer I would expect in an interview
A strong engineer can describe a reproducible bad query, show how the index and analyzer generated candidates, explain what BM25 optimized, and produce a relevance dataset before proposing changes. They can identify the hard filters, budget candidate depth, and evaluate whether a second-stage ranker actually lifts useful results.
The central distinction is straightforward. Fast retrieval, good lexical matching, useful ranking and business success are four different measurements. A mature search system tracks all four because improving one can damage another. I'd ship a carefully tested query and ranking change before I bought more hardware to accelerate a relevance mistake.
