Recommenders · Ranking · Evaluation

Where do recommender wins actually come from?

I built a two-stage recommender on MovieLens-1M. The retriever alone took NDCG@10 to 2.3 times the popularity baseline. The second stage on top (extra candidate sources plus a learned ranker) added another 12%, and cost 67 times the median latency of retrieval alone. Then I asked what the ranker had learned, and its most important feature appears to be partly a fingerprint of how I had built its training data.

The question

Once a retriever gives you a few hundred good candidates, how much does a second-stage ranker add, and is that worth what it costs to serve?

The setup

Code and saved outputs: streamrec. Training and sampling are seeded (SEED = 42).

The result: retrieval does the heavy lifting

NDCG@10 on the test split, 6,040 users Popularity 0.018, SASRec 0.029, two-tower 0.042, two-stage retrieve plus rank 0.046, which is 12% above the two-tower. 0 0.01 0.02 0.03 0.04 0.05 NDCG@10 on the held-out test item (higher is better) Popularity baseline Popularity: NDCG@10 0.0180 0.018 SASRec retrieval only SASRec: NDCG@10 0.0293 0.029 Two-tower retrieval only Two-tower + HNSW: NDCG@10 0.0415, 2.3x popularity 0.042 Two-stage retrieve + rank Two-stage (retrieve + LambdaMART): NDCG@10 0.0464, +12% over two-tower 0.046 (+12%) NDCG@10 on the test split, 6,040 users Popularity 0.018, SASRec 0.029, two-tower 0.042, two-stage retrieve plus rank 0.046, which is 12% above the two-tower. 0 0.02 0.04 0.05 NDCG@10 (higher is better) Popularity (baseline) Popularity: NDCG@10 0.0180 0.018 SASRec (retrieval only) SASRec: NDCG@10 0.0293 0.029 Two-tower (retrieval only) Two-tower + HNSW: NDCG@10 0.0415, 2.3x popularity 0.042 Two-stage (retrieve + rank), +12% Two-stage (retrieve + LambdaMART): NDCG@10 0.0464, +12% over two-tower 0.046
NDCG@10 on each user's last rating, ranked against the full catalogue. Blue bars are single retrievers; the orange bar is the full retrieve-then-rank pipeline. Hover a bar for its value.
Chart data as a table
ModelNDCG@10Recall@10Coverage@20
Popularity0.01800.03690.08
SASRec0.02930.05810.10
Two-tower + HNSW0.04150.08480.74
Two-stage (retrieve + rank)0.04640.08680.57

Of the whole gain from popularity to the full pipeline, the two-tower retriever accounts for roughly 80% and the second stage (extra candidate sources plus the ranker) for roughly 20%. I cannot split that 20% between the two: the missing ablation is a fusion of the retrievers with no learned ranker, such as reciprocal rank fusion. Recall@10 barely moves (0.085 to 0.087, about 12 users out of 6,040), so the gain is reordering, not finding new items. Coverage in the top 20 also falls from 0.74 to 0.57.

Path (CPU, 300 users, k = 10)p50p95p99
Retrieval onlytwo-tower lookup, one HNSW query0.11 ms0.29 ms0.42 ms
Two-stagetwo-tower top 200, SASRec top 200, popularity fill and merge, features, LightGBM7.4 ms9.0 ms9.5 ms
Ratio67x31x23x

Both paths ran in the same process on the same users, so the numbers are comparable, though not tightly controlled: the paths ran one block after the other, not interleaved, with no warm-up. And the 7.4 ms is not "the ranker". It also pays for SASRec, which runs a transformer over the user's last 50 or fewer items and scores every movie against that state with one matrix multiply, plus the popularity fill, the merge, and a Python loop building a feature row per candidate. I did not time the stages separately. The multiple also shrinks in the tail, to 23x at p99.

The surprise: the ranker's top feature was partly a training artifact

By LightGBM gain, n_sources is the ranker's most important feature: 42% of total gain, 3.4 times the next, item popularity (about 2.4 times if you add item_pop_rank, since popularity is split across related columns). The tidy story: two retrievers with different biases agreeing on an item is good evidence, and the ranker learned to trust agreement.

Then I read the code that builds the training groups. To guarantee each group has a positive, it inserts the user's validation item into the pool when neither retriever found it. An inserted item gets n_sources = 0, zero retriever scores and the worst ranks. The only other candidates with n_sources = 0 are popularity fill-ins that neither retriever surfaced, all among the 50 most popular unseen movies. So within one training group, "no retriever found it and it is not top-50 popular" means "this is the answer", exactly. n_sources plus popularity is sufficient for a tree to express that, though not unique: the zero scores and worst ranks mark the item just as well. At serving time nothing is inserted. How often insertion happened was not saved, so the size of the effect is an inference.

The saved outputs suggest the ranker learned it. The bandit replay script also inserts the held-out test item. There, the ranker's top pick was the held-out item in 108 of 278 replayed events (38.8%); the 278 are a random one-in-twenty subsample, the users whose logged arm was the top slot. When the item was retrieved, nothing is inserted and the pool matches the honest evaluation's, apart from a train-only seen set. There the item reaches the top 5 for only 5.4% of users, so top-1 is at most 5.4%. At least 33 points of the 38.8% (about 28 at the low end of a 95% interval) therefore come from inserted items with zero scores and worst ranks, which a sensible ranker would put last. Its effect on the gain figures stays an inference until I retrain without insertion.

The test NDCG is still honest: nothing is inserted at test time. What breaks is the explanation. "Retriever agreement is the strongest signal" is not established, and LinUCB against greedy (0.387 vs 0.388) was measured on a log with the answer planted, so it says nothing about the bandit. With standard errors near 0.03, the two are within noise anyway.

The root cause is ordinary train/serve skew, moved into candidate generation: training rows must come from the distribution the ranker will score.

What I would do in production

Provisional decision: retrieval only on latency-critical user-to-item surfaces, where every millisecond is felt and the second stage adds 12%; the two-stage pipeline for the homepage row, where 9.5 ms at p99 buys better ordering of the first thing a user sees. (Search and item-to-item "more like this" are different problems; I did not measure them here.)

Before quoting the 12% to anyone, I would make three changes. Build training groups exactly as serving does, dropping users whose positive was not retrieved: fewer positives, but each one teaches the real problem. Compare against reciprocal rank fusion of the two retrievers; if it recovers most of the 12%, the learned ranker is not paying its way. And time each stage, so the latency lands on the component that causes it.

Limits

Code and saved outputs: github.com/trohit920/streamrec. Previous post: Leading an AI-augmented build.

← All posts