Being accurate is not enough: how accuracy metrics have hurt recommender systems

By Sean M. McNee, John Riedl, J. Konstan. CHI Extended Abstracts, 2006

1,237 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/e84e9f82f49eb5a21cdf2306e42478773ff9e82a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Evaluating the effectiveness of explanations for recommender systems (2012, background): “Sometimes when adaptive systems are evaluated, there is a shortage of information about what exactly they are being evaluated on (McNee et al. 2006b).”
Short and tweet: experiments on recommending content from information streams (2010, background): “The issue of serendipity has been raised in both web search [3] and recommender systems [11].”
News recommender systems - Survey and roads ahead (2018, background): “This recent trend can be considered as a positive development, since it has already been shown in other domains in which recommenders have been applied that focusing solely on accuracy measures to compare algorithms can be insufficient and potentially misleading [80, 128].”
Music recommendation and discovery in the long tail (2008, background): “Without any user intervention it is impossible to evaluate the quality and user satisfaction of the recommendations, which does not necessarily correlate with predicted accuracy (McNee et al., 2006).”
Captivating algorithms: Recommender systems as traps (2018, background): “As the title of one influential early paper put it, ‘being accurate is not enough’ (McNee et al., 2006).”
User Preferences for Hybrid Explanations (2017, background): “E ective recommendations must be both accurate and explainable [9, 10, 13, 16].”
Evaluating Recommender Systems with User Experiments (2015, background): “[83] who argued that “being accurate is not enough” and that one should instead “study recommenders from a user-centric perspective to make them not only accurate and helpful, but also a pleasure to use” (p.”
Data Mining Methods for Recommender Systems (2015, background): “point out [51], there is much more than accuracy”
The effects of recommendations’ presentation on persuasion and satisfaction in a movie recommender system (2010, background): “But still, computational accuracy is not enough to create useful or credible recommendations [19].”
A cross-benchmark comparison of 87 learning to rank methods (2015, background): “In addition, McNee, Riedl, and Konstan (2006) found the ranking task to be a better ﬁt for recommender systems than the regression task (continuous scale predictions), which is currently still frequently used within such systems.”
Designing and Evaluating Explanations for Recommender Systems (2011, background): “In recent years, there has been an increased interest in more user-centered evaluation metrics for recommender systems such as those mentioned in [42].”
Explaining Recommendations: Design and Evaluation (2015, background): “In recent years, there has been an increased interest in more user-centered evaluation metrics for recommender systems such as those mentioned in [49].”
Deep Learning Based Recommender System (2017, background): “However, being accurate is far from enough for a high-quality recommender system in the long run [73].”
Explanations of recommendations (2007, background): “The recommender systems community is reaching a consensus that accuracy metrics such as mean average error (MAE), precision and recall, can only partially evaluate a recommender system [15].”