Estimating the Causal Impact of Recommendation Systems from Observational Data

By Amit Sharma, Jake M. Hofman, Duncan J. Watts. 2015

112 citations recorded by the source corpus.

Recommendation systems are an increasingly prominent part of the web, accounting for up to a third of all traffic on several of the world's most popular sites. Nevertheless, little is known about how much activity such systems actually cause over and above activity that would have occurred via other means (e.g., search) if recommendations were absent. Although the ideal way to estimate the causal impact of recommendations is via randomized experiments, such experiments are costly and may inconvenience users. In this paper, therefore, we present a method for estimating causal effects from purely observational data. Specifically, we show that causal identification through an instrumental variable is possible when a product experiences an instantaneous shock in direct traffic and the products recommended next to it do not. We then apply our method to browsing logs containing anonymized activity for 2.1 million users on Amazon.com over a 9 month period and analyze over 4,000 unique products that experience such shocks. We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations. We conclude with a discussion about the assumptions under which the method is appropriate and caveats around extrapolating results to other products, sites, or settings.

Subfields: Computer Science Applications, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, methodology): “…bias, under strong assumptions; including matched analysis (De Choudhury et al., 2016; Sharma and Cosley, 2016), instrumental variables analysis (Sharma et al., 2015), regression discontinuities (Malik and Pfeﬀer, 2016), and diﬀerences-in-diﬀerences (Carmi et al., 2012; Zagheni et al., 2014).”
How algorithmic confounding in recommendation systems increases homogeneity and decreases utility (2017, background): “Some work has also been done to understand the causal impact of these systems on behavior by nding natural experiments in observational data [53, 55] (approximating expensive controlled experiments [33]), but it is unclear howwell these results generalize.”
The Limits of Popularity-Based Recommendations, and the Role of Social Ties (2016, unknown): “Note that this is a conservative estimate, since the available evidence suggests both the relative inﬂuence of friends’ recommendations (see e.g. [20, 31]) and the fraction of sales causally imputable to recommender systems (see [27]) to be signiﬁcantly smaller.”
The Effect of Recommendations on Network Structure (2016, background): “However, as has been noted previously [25], such an estimation scheme can lead to spurious results.”