Evaluation of recommendations: rating-prediction and ranking

By Harald Steck. 2013

280 citations recorded by the source corpus.

The literature on recommender systems distinguishes typically between two broad categories of measuring recommendation accuracy: rating prediction, often quantified in terms of the root mean square error (RMSE), and ranking, measured in terms of metrics like precision and recall, among others. In this paper, we examine both approaches in detail, and find that the dominating difference lies instead in the training and test data considered: rating prediction is concerned with only the observed ratings, while ranking typically accounts for all items in the collection, whether the user has rated them or not. Furthermore, we show that predicting observed ratings, while popular in the literature, only solves a (small) part of the rating prediction task for any item in the collection, which is a common real-world problem. The reasons are selection bias in the data, combined with data sparsity. We show that the latter rating-prediction task involves the prediction task 'Who rated What' as a sub-problem, which can be cast as a classification or ranking problem. This suggests that solving the ranking problem is not only valuable by itself, but also for predicting the rating value of any item.

Subfields: Information Systems, Management Science and Operations Research, Marketing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

In Marketing

Citations in the corpus

How algorithmic confounding in recommendation systems increases homogeneity and decreases utility (2017, background): “Low errors for predicting ratings, however, do not always correspond to high accuracy in rank-based evaluation [14, 54] (e.”
Recommendations as Treatments: Debiasing Learning and Evaluation (2016, background): “More generally, under selection bias, ˆ R naive ( ˆ Y ) is not an unbiased estimate of the true performance R ( ˆ Y ) (Steck, 2013): Before we design an improved estimator to replace ˆ R naive ( ˆ Y ) , let’s turn to a related evaluation task.”
Top-N Recommendation with Missing Implicit Feedback (2015, background): “Our work is most closely related to [8, 9].”
Advances and Challenges in Conversational Recommender Systems: A Survey (2021, background): “[66] analyze the phenomenon of popularity bias [156, 133] and selection bias [113, 61, 157] in simulators built on logged interaction data and try to alleviate model performance degradation due to these biases; it remains to be seen to which degree generated interactions of the type described above ”
Bias and Debias in Recommender System: A Survey and Future Directions (2020, methodology): “The distribution of observed rating data is different from the distribution of all ratings [4], [21].”
entity2rec: Learning User-Item Relatedness from Knowledge Graphs for Top-N Item Recommendation (2017, background): “evaluation protocol allows to avoid the limitations of the TestItems protocol, in which only rated items are ranked [26] and that is thus very dierent from the context of a real recommender system.”
Computing Recommendations via a Knowledge Graph-aware Autoencoder (2018, methodology): “For the evaluation of our approach we adopted the "all unrated items" protocol described in [41]: for each user u, a top-N recommendation list is provided by computing a score for every item i not rated by u, whether i appears in the user test set or not.”
Recommender Systems (2016, unknown): “An approach that has been used in recent years, is to correct for this bias by modeling the bias in the missing rating distribution within the evaluation measures [565, 566].”
The use of machine learning algorithms in recommender systems: A systematic review (2015, background): “Today, RSs are used in many information-based companies such as Google [35], Twitter [4], LinkedIn [50], and Netflix [59].”