On Sampled Metrics for Item Recommendation
By Walid Krichene, Steffen Rendle. Knowledge Discovery and Data Mining, 2020
623 citations recorded by the source corpus.
The task of item recommendation requires ranking a large catalogue of items given a context. Item recommendation algorithms are evaluated using ranking metrics that depend on the positions of relevant items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates sampled metrics in more detail and shows that they are inconsistent with their exact version, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation. Moreover, the smaller the sampling size, the less difference there is between metrics, and for very small sampling size, all metrics collapse to the AUC metric. We show that it is possible to improve the quality of the sampled metrics by applying a correction, obtained by minimizing different criteria such as bias or mean squared error. We conclude with an empirical evaluation of the naive sampled metrics and their corrected variants. To summarize, our work suggests that sampling should be avoided for metric calculation, however if an experimental study needs to sample, the proposed corrections can improve the quality of the estimate.
Subfields: Information Systems, Management Science and Operations Research
View Paper: https://www.semanticscholar.org/paper/41f9fd1c997473d8c50a4f7f61eb317ac7a6f25c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
In Management Science and Operations Research
Citations in the corpus
- A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation (2021, background): “For example, researchers argue that previously default choice of evaluating recommender models with sampled metrics (e.g., rather than using the full set, only sampling a small set of negative items during testing) would be inconsistent to the true trend [279].”
- Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2021, methodology): “We evaluate the ranking results over the whole item set for the fair comparison [22].”
- One Person, One Model, One World: Learning Continual User Representation without Forgetting (2020, methodology): “sult of this final epoch in the following epoches. item recommendation tasks, we evaluate MRR@5 using its exact version rather than the sampled version as used in [42]. We refer interested readers to [18] for more details. ComparedMethods.To the best of our knowledge, there is no existing user model”
- Advances and Challenges in Conversational Recommender Systems: A Survey (2021, background): “However, Krichene and Rendle [92] point out and prove that some metrics, such as average precision, recall, and NDCG, are inconsistent with the exact metrics when they are calculated on the sampled set.”
- RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms (2020, background): “We also consider both full ranking and sampling-based ranking, which is recently a controversial issue in the field of recommender system [32].”
- Intent Contrastive Learning for Sequential Recommendation (2022, positive): “We follow [16, 42] to rank the prediction on the whole item set without negative sampling.”
- Learning Intents behind Interactions with Knowledge Graph for Recommendation (2021, methodology): “To evaluate topK recommendation, we adopt the protocols [20]: recall@K and ndcg@K , where K is set as 20 by default.”
- Contrastive Learning for Sequential Recommendation (2022, methodology): “However, this sample operation may lead to inconsistent with their non-sampled version as illustrated by Krichene and Rendle [26].”