Explaining the user experience of recommender systems

By Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, Chris Newell. 2012

879 citations recorded by the source corpus.

Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.

Subfields: Applied Psychology, General Decision Sciences, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In General Decision Sciences

In Information Systems

Citations in the corpus

Explainable Recommendation: A Survey and New Perspectives (2018, background): “, 2009), satisfaction (Bilgic and Mooney, 2005), and scrutability of the system (Balog et al., 2019; Knijnenburg et al., 2012).”
Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity (2018, neutral): “form the field of computer science with psychology have repeatedly shown that diversity in recommendation sets increases user satisfaction (Knijnenburg et al., 2012; Ziegler et al., 2005; Willemsen et al., 2016).”
Short-Term Satisfaction and Long-Term Coverage: Understanding How Users Tolerate Algorithmic Exploration (2018, methodology): “We mainly follow the user-centric paradigm [26, 30, 36] for the evaluation of recommender systems in this paper.”
Recommender Systems: Introduction and Challenges (2015, background): “The chapter first presents a theoretical user-centric evaluation framework [29] that maps aspects of recom-”
Collaborative Filtering beyond the User-Item Matrix (2014, background): “Explanations provide the users with rationale that motivates why the items/products have been recommended [Friedrich and Zanker 2011 ; Herlocker et al. 2000; Knijnenburg et al. 2012; Symeonidis et al. 2008a].”
Diversity, Serendipity, Novelty, and Coverage (2016, background): “…between the different recommendation qualities perceived by users (e.g., the impact of perceived novelty on diversity perception) [Pu et al. 2011; Knijnenburg et al. 2012; Ekstrand et al. 2014], and (2) works that analyze the impact of the proposed algorithms (or user interface modifications) on…”
Recommender systems: from algorithms to user experience (2012, background): “Our review looks at research grounded in specific recommender systems and their evaluations, and stands in contrast to Knijnenburg et al. (2012) which approaches user experience from more of an experience-model and social-experimental approach.”
Recommender systems based on user reviews: the state of the art (2015, methodology): “In designing and implementing user evaluation, the recently developed user-centered evaluation frameworks for RS could be referential (Knijnenburg et al. 2012; Pu et al. 2011).”
Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols (2013, methodology): “2011), and situational and personal characteristics of users (Knijnenburg et al. 2012) may be related with such differences.”
A Survey on Conversational Recommender Systems (2020, background): “, those discussed in the evaluation frameworks in [108] and [66].”
Inspectability and control in social recommenders (2012, methodology): “Users seem to appreciate control in their interaction with recommender systems [28, 33].”
News recommender systems - Survey and roads ahead (2018, background): “In recent years, however, recommender systems research in a variety of domains has shown that it is also important to evaluate recommendations in a more user- or system utility-oriented way [48, 78, 83, 96, 151].”
Counteracting the Negative Effect of Form Auto-completion on the Privacy Calculus (2013, methodology): “…2013 Finally, we measured satisfaction with the FormFiller tool using a scale (Alpha: .93, AVE: .791) adapted from (Knijnenburg and Kobsa 2013a; Knijnenburg et al. 2012), with the following 6 items, measured on a 7-point scale from “completely disagree” to “completely agree”: • “I would use…”
Captivating algorithms: Recommender systems as traps (2018, background): “New ‘user-centered’ studies that sought to measure satisfaction through survey instruments found a striking result: beyond a certain point, improvements in RMSE did not correlate with increased user satisfaction (Knijnenburg et al., 2012; see also Pu et al., 2011).”
A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation (2013, background): “Typically, offline evaluations are meant to identify the most promising recommendation approaches [5,6,19].”
Evaluating Recommender Systems with User Experiments (2015, methodology): “In fact, non-personalized diversified lists can be as attractive as personalized recommendations [67].”
Research-paper recommender systems: a literature survey (2015, background): “with real users" [284].”
Evaluating recommender systems from the user’s perspective: survey of the state of the art (2012, methodology): “In comparison with Knijnenburg et al. (2012), the ResQue framework and its outcome were validated independently of the underlying recommender systems or the algorithms.”
Automatic Generation of Natural Language Explanations (2017, background): “Furthermore, this source plays an important role on the user side, since he/she tend to trust the opinion of other users [5, 13, 22].”