Evaluating collaborative filtering recommender systems

By Jonathan L. Herlocker, J. Konstan, Loreen Terveen, John Riedl. 2004

6,139 citations recorded by the source corpus.

Subfields: Information Systems, Management Science and Operations Research, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

How algorithmic confounding in recommendation systems increases homogeneity and decreases utility (2017, background): “Accuracy metrics are popular because they are straightforward to understand and easy to implement, but still they do not necessarily capture the usefulness of recommendations; there are a variety of system characteristics that are thought to be important, along with methods for evaluating them [24].”
Efficient Private Statistics with Succinct Sketches (2015, neutral): “These recommendations are usually provided by means of recommender systems [3, 41] like ItemKNN (cf. Section II-D), which typically require the full availability of users’ ratings, whereas, we focus on a model where a provider like the BBC provides recommendations to its users, e.g., on iPlayer,…”
Novelty and Diversity in Top-N Recommendation -- Analysis and Evaluation (2011, methodology): “The accuracy metrics typically used to measure system performance are largely directed at average performance over a large number of test users, such as Mean Absolute Error (MAE) [Herlocker et al. 2004], or precision and recall [Manning et al.”
A Comprehensive Survey of Neighborhood-based Recommendation Methods (2011, background): “[24, 26], the optimal value of k should be determined by cross-validation.”
Link Prediction in Complex Networks: A Survey (2010, methodology): “For Precision, if L = 2, the predicted links are (1 , 4) and (4 , 5).”
A survey of collaborative filtering based social recommender systems (2014, background): “Since most of the existing social recommender systems are CF-based, we focus on CF-based traditional RSs in this section.”
Evaluation of contextual information retrieval effectiveness: overview of issues and research (2010, methodology): “2.1. Furthermore, we outline that regarding the sub-dimension of the user context, we only address the individual user in a retrieval setting (reactive approach valuable using a IRS), not the group or community of users in a filtering setting (proactive approach valuable in a filtering or recommende”
Social contextual recommendation (2012, methodology): “We design our experiments based on two user tasks [9]: (1) annotation in context (2) find good items.”
Novelty and Diversity in Recommender Systems (2015, background): “While most research in the Recommender Systems has focused on accuracy in matching user interests, there is increasing consensus in the community that accuracy alone is not enough to assess the practical effectiveness and added-value of recommendations [12,16].”
Combining content-based and collaborative recommendations: A hybrid approach based on Bayesian networks (2010, methodology): “Following [45], we propose to use the mean absolute error (MAE) which measures how close system predictions are to the user’s rating for each movie by considering the average absolute deviation between a predicted rating and the user’s true rating.”
Precision-oriented evaluation of recommender systems: an algorithmic comparison (2011, background): “In fact, as discussed in [3], recall depends heavily on the number of relevant items that each user has rated, and therefore, it should only be used for comparison purposes, not interpreted as an absolute measure.”
Short-Term Satisfaction and Long-Term Coverage: Understanding How Users Tolerate Algorithmic Exploration (2018, methodology): “This task is an instance of what is often referred to as the Find Good Items task [18], or One-Choice task [34].”
Integro: Leveraging Victim Prediction for Robust Fake Account Detection in OSNs (2015, methodology): “This percentage is directly related to the precision of fake account detection, which is a performance metric typically used to measure the ratio of relevant items over the top-k highest ranked items in terms of relevance [61].”
Empirical Analysis of the Impact of Recommender Systems on Sales (2010, methodology): “In these studies, real preference data of customers are obtained from surveys or controlled field experiments and are then compared to the recommendations produced by various algorithms and systems [21, 23, 42].”
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks (2009, background): “Providing a single definition for recommender systems is difficult, mainly becausesyst ms with different objectives and behaviors are grouped together under thatname.”