Abi Abiassi
WRITING /
The model was fine. Retrieval wasn't.
A scorer can only rank what retrieval gives it.
In March, we worked on improving a product-matching scorer for a competitive-intelligence product built for a major global fashion retailer.
The job looked simple from a distance. Take one product from the internal catalog, find comparable products sold by competitors, and put the best matches first. An analyst should be able to inspect the top three and trust what they see.
The first scorer did not earn that trust. Visually similar products from different categories could rank highly. A bracelet and an earring might share color, material, and a strong image score while still being useless as competitive alternatives. At the same time, a correct match with sparse name or category overlap could be buried.
We needed a number that noticed the difference.
The number behind the ranking
We used nDCG@3, or Normalized Discounted Cumulative Gain at three results.
The name is worse than the idea. Give every candidate a relevance grade. Reward high grades. Discount that reward as the candidate moves down the list. Then divide by the score of the ideal ordering, so every query lands between zero and one.
A perfect result at position one should count more than the same result at position three. A strong match should count more than several marginal ones. nDCG captures both.
The metric came from information-retrieval research by Kalervo Järvelin and Jaana Kekäläinen. They introduced cumulative and discounted gain at SIGIR in 2000, then formalized normalized DCG in 2002. Precision and recall treated relevance as binary and largely ignored whether the best document appeared early. Their work added graded relevance and rank discounting.
That made nDCG a natural fit for web search. It became standard across TREC web tracks and learning-to-rank benchmarks. RankNet arrived in 2005. LambdaRank followed in 2006, engineering gradients around rank swaps because nDCG itself depends on a sort and cannot be optimized directly. LambdaMART later became the dominant industrial learning-to-rank model and a fixture on web-search benchmarks.
Now the judgments themselves can come from language models. TREC's 2024 RAG Track used LLM relevance assessors at scale. Small teams can build judged sets without maintaining a human rater pool, although the evidence still calls for human calibration and warns that LLM judges tend to overrate marginal results.
Our set was LLM-judged: 1,688 product pairs across 219 query groups, labeled correct, partial, or incorrect. We used @3 because the product surfaced three candidates. Anything below that was outside the analyst's immediate decision surface.
This choice had a blind spot. nDCG@3 could tell us whether retrieved candidates were ordered well. It could not tell us whether the right candidate entered the pool at all.
That mattered later.
Fifty experiments and a simple scorer
The evaluation harness was deliberately fixed. The experiment loop could change the scorer, run it against the judged set, log the result, and keep an improvement. It could not change the metric definition.
More than 50 experiments were logged across the research waves. Early formula tweaks moved the score by fractions of a point. Then the search widened.
Logistic regression lost. LightGBM LambdaRank lost. Cross-encoders were tried three times and lost each time. Each challenger was measured against the current simple scorer on the same judged evaluation set for that experiment. Complexity did not get credit for being interesting.
The scorer that shipped was an 11-signal log-linear composite. It multiplied signals in log space, with weights optimized by differential evolution. The useful change paired that search with a floor for each signal.
Without floors, one signal at exactly zero could collapse the full multiplicative score, even when the other signals described a good match. The floors stopped that failure. Visual similarity, which had encouraged attractive but wrong-type matches, ended up with very little weight. Name overlap became much more important.
On the committed March evaluation set, judged matching quality moved from 0.8446 to 0.9209 nDCG@3. The production scorer used the 11-signal log-linear design with per-signal floors. Later routing experiments ran on a separate, rebalanced set, but the evidence does not show that version reaching production.
At that point the ranking work had done its job. Given candidates, the scorer was good at putting the useful ones near the top.
Three months later, a different failure
In June, retrieval coverage was stuck at 28.7 percent. This metric asked a different question: what fraction of internal products received any competitor match at all?
The cause sat before the scorer.
Candidate generation searched an enriched pool of 86,440 competitor products. Only 8,936 were active, roughly ten percent. For each internal product, the retrieval pass had a budget of 50 nearest candidates. Most of that budget could be consumed by inactive products. A later join dropped them before scoring.
The scorer never saw the missing active candidates. It could not rank what retrieval had removed.
The fix was to build the candidate search over the active, enriched pool by default. Because every product could now find some nearest neighbor, two quality controls mattered as well: a higher composite-score floor and a product-type distance gate to reject cross-category noise.
On staging, retrieval coverage moved from 28.7 percent to 78.5 percent. At the time of the commits, the production cron had not yet switched over, so this was a staging-verified result.
nDCG made scorer changes falsifiable. It could not tell us whether retrieval had starved the scorer.
Nothing about the March improvement caused the June gain. Different stage. Different metric. Different fix.