Where do recommender wins actually come from?
I built a two-stage recommender on MovieLens-1M. The retriever alone took NDCG@10 to 2.3 times the popularity baseline. The second stage on top (extra candidate sources plus a learned ranker) added another 12%, and cost 67 times the median latency of retrieval alone. Then I asked what the ranker had learned, and its most important feature appears to be partly a fingerprint of how I had built its training data.
The question
Once a retriever gives you a few hundred good candidates, how much does a second-stage ranker add, and is that worth what it costs to serve?
The setup
- Data: MovieLens-1M, split per user by timestamp: last rating is test, second-last is validation, the rest is training. All 6,040 users are evaluated on one held-out item, ranked against the whole catalogue minus what they have rated. No sampled negatives.
- Retrievers: popularity as the baseline; a two-tower model (64-dimensional user and item embeddings, in-batch softmax with a logQ popularity correction, served from an HNSW index); and a small SASRec-style sequence model.
- Ranker: LightGBM LambdaMART (300 trees) over the two-tower's top 200 and SASRec's top 200 (each minus already-rated items) plus the 50 most popular unseen items. Twelve features, including each retriever's score and rank, item and user statistics, and
n_sources: how many of the two retrievers surfaced the item. Trained on the validation item of the first 4,000 users by ID. - Latency: single requests, one after another, for the same 300 sampled users on CPU, asking for 10 items.
Code and saved outputs: streamrec. Training and sampling are seeded (SEED = 42).
The result: retrieval does the heavy lifting
Chart data as a table
| Model | NDCG@10 | Recall@10 | Coverage@20 |
|---|---|---|---|
| Popularity | 0.0180 | 0.0369 | 0.08 |
| SASRec | 0.0293 | 0.0581 | 0.10 |
| Two-tower + HNSW | 0.0415 | 0.0848 | 0.74 |
| Two-stage (retrieve + rank) | 0.0464 | 0.0868 | 0.57 |
Of the whole gain from popularity to the full pipeline, the two-tower retriever accounts for roughly 80% and the second stage (extra candidate sources plus the ranker) for roughly 20%. I cannot split that 20% between the two: the missing ablation is a fusion of the retrievers with no learned ranker, such as reciprocal rank fusion. Recall@10 barely moves (0.085 to 0.087, about 12 users out of 6,040), so the gain is reordering, not finding new items. Coverage in the top 20 also falls from 0.74 to 0.57.
| Path (CPU, 300 users, k = 10) | p50 | p95 | p99 |
|---|---|---|---|
| Retrieval onlytwo-tower lookup, one HNSW query | 0.11 ms | 0.29 ms | 0.42 ms |
| Two-stagetwo-tower top 200, SASRec top 200, popularity fill and merge, features, LightGBM | 7.4 ms | 9.0 ms | 9.5 ms |
| Ratio | 67x | 31x | 23x |
Both paths ran in the same process on the same users, so the numbers are comparable, though not tightly controlled: the paths ran one block after the other, not interleaved, with no warm-up. And the 7.4 ms is not "the ranker". It also pays for SASRec, which runs a transformer over the user's last 50 or fewer items and scores every movie against that state with one matrix multiply, plus the popularity fill, the merge, and a Python loop building a feature row per candidate. I did not time the stages separately. The multiple also shrinks in the tail, to 23x at p99.
The surprise: the ranker's top feature was partly a training artifact
By LightGBM gain, n_sources is the ranker's most important feature: 42% of total gain, 3.4 times the next, item popularity (about 2.4 times if you add item_pop_rank, since popularity is split across related columns). The tidy story: two retrievers with different biases agreeing on an item is good evidence, and the ranker learned to trust agreement.
Then I read the code that builds the training groups. To guarantee each group has a positive, it inserts the user's validation item into the pool when neither retriever found it. An inserted item gets n_sources = 0, zero retriever scores and the worst ranks. The only other candidates with n_sources = 0 are popularity fill-ins that neither retriever surfaced, all among the 50 most popular unseen movies. So within one training group, "no retriever found it and it is not top-50 popular" means "this is the answer", exactly. n_sources plus popularity is sufficient for a tree to express that, though not unique: the zero scores and worst ranks mark the item just as well. At serving time nothing is inserted. How often insertion happened was not saved, so the size of the effect is an inference.
The saved outputs suggest the ranker learned it. The bandit replay script also inserts the held-out test item. There, the ranker's top pick was the held-out item in 108 of 278 replayed events (38.8%); the 278 are a random one-in-twenty subsample, the users whose logged arm was the top slot. When the item was retrieved, nothing is inserted and the pool matches the honest evaluation's, apart from a train-only seen set. There the item reaches the top 5 for only 5.4% of users, so top-1 is at most 5.4%. At least 33 points of the 38.8% (about 28 at the low end of a 95% interval) therefore come from inserted items with zero scores and worst ranks, which a sensible ranker would put last. Its effect on the gain figures stays an inference until I retrain without insertion.
The test NDCG is still honest: nothing is inserted at test time. What breaks is the explanation. "Retriever agreement is the strongest signal" is not established, and LinUCB against greedy (0.387 vs 0.388) was measured on a log with the answer planted, so it says nothing about the bandit. With standard errors near 0.03, the two are within noise anyway.
The root cause is ordinary train/serve skew, moved into candidate generation: training rows must come from the distribution the ranker will score.
What I would do in production
Provisional decision: retrieval only on latency-critical user-to-item surfaces, where every millisecond is felt and the second stage adds 12%; the two-stage pipeline for the homepage row, where 9.5 ms at p99 buys better ordering of the first thing a user sees. (Search and item-to-item "more like this" are different problems; I did not measure them here.)
Before quoting the 12% to anyone, I would make three changes. Build training groups exactly as serving does, dropping users whose positive was not retrieved: fewer positives, but each one teaches the real problem. Compare against reciprocal rank fusion of the two retrievers; if it recovers most of the 12%, the learned ranker is not paying its way. And time each stage, so the latency lands on the component that causes it.
Limits
- Offline, one dataset. MovieLens-1M ratings, not views or plays; every rating counts as a positive, including one-star ones; one held-out item per user.
- No confidence intervals. The per-user scores are not saved. An upper bound on each mean's standard error is about 0.0027, so the 0.005 NDCG gap is about 1.3 unpaired standard errors. I would want a paired test before trusting its size.
- One machine. The CPU is not recorded; no batching or concurrency.
- Not every README claim is in the saved outputs. The logQ correction more than doubling Recall@10 is per the README; the ablation output was not saved, so it is not reproducible from saved outputs.
Code and saved outputs: github.com/trohit920/streamrec. Previous post: Leading an AI-augmented build.