Recommendations as Treatments: Debiasing Learning and Evaluation
By Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, Thorsten Joachims. 2016
833 citations recorded by the source corpus.
Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself. In this paper, we provide a principled approach to handle selection biases by adapting models and estimation techniques from causal inference. The approach leads to unbiased performance estimators despite biased data, and to a matrix factorization method that provides substantially improved prediction performance on real-world data. We theoretically and empirically characterize the robustness of the approach, and find that it is highly practical and scalable.
Subfields: Information Systems, Management Science and Operations Research, Statistics and Probability, Recommender Systems, AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Xiangnan He
- Xiang Wang
- Fuli Feng
- Meng Wang
- Yong Li
- Chen Gao
- Thorsten Joachims
- Mihaela van der Schaar
In Management Science and Operations Research
- Thorsten Joachims
- Xiangnan He
- Mihaela van der Schaar
- Nathan Kallus
- Adith Swaminathan
- Alekh Agarwal
- James Jordon
- Alberto Bietti
In Statistics and Probability
- Mihaela van der Schaar
- James Jordon
- Thorsten Joachims
- Jinsung Yoon
- Ahmed M. Alaa
- Adarsh Subbaswamy
- Suchi Saria
- Alekh Agarwal
In Recommender Systems
- Thorsten Joachims
- Mihaela van der Schaar
- Jinsung Yoon
- Xiang Wang
- Adith Swaminathan
- Fuli Feng
- Alekh Agarwal
- Xiangnan He
In AI Ethics, Fairness & Society
- Mihaela van der Schaar
- Jinsung Yoon
- Thorsten Joachims
- Aram Galstyan
- Kangwook Lee
- Fred Morstatter
- Ninareh Mehrabi
- Michael Gira
Citations in the corpus
- A Survey on Bias and Fairness in Machine Learning (2019, background): “Bias in facial recognition systems [106] and recommender systems [114] have also been largely studied and evaluated and in many cases shown to be discriminative towards certain populations and subgroups.”
- How algorithmic confounding in recommendation systems increases homogeneity and decreases utility (2017, methodology): “[52] use propensity weighting techniques to remove users’ selection bias for explicit ratings.”
- GAIN: Missing Data Imputation using Generative Adversarial Nets (2018, methodology): “…2000; Buuren & Groothuis-Oudshoorn, 2011), MissForest (Stekhoven & B ¨ uhlmann, 2011), and matrix completion (Mazumder et al., 2010a; Yu et al., 2016; Schnabel et al., 2016; Mazumder et al., 2010b); generative methods include algorithms based on Expectation Maximization (Garc´ıa-Laencina et al.,…”
- Causal embeddings for recommendation (2017, methodology): “A set of new class of embedding-based approaches that leverage this setup have been recently proposed in (Johansson, Shalit, and Sontag 2016; Shalit, Johansson, and Sontag 2016) and the closely related work in (Rosenfeld, Mansour, and Yom-Tov 2016).”
- Learning Treatment-Response Models from Multivariate Longitudinal Data (2017, methodology): “On simulated and a challenging clinical dataset, we show significant gains in accuracy over stateof-the-art models.”
- Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks (2017, methodology): “Matrix completion methods such as [7, 8, 9, 10] do exploit correlations within and across streams, but assume that the data is static – hence ignore the temporal component of the data – or that the data is perfectly synchronized – an assumption that is routinely violated in medical time series data.”
- A Contextual Bandit Bake-off (2018, positive): “Such an estimator has been used, e.g. , in the context of off-policy learning for recommendations (Schnabel et al., 2016) and is available in the Vowpal Wabbit library.”
- Lessons from Real-World Reinforcement Learning in a Customer Support Bot (2019, background): “This is the case of many user-facing systems, such as news (Li et al., 2010) and video (Schnabel et al., 2016; Chen et al., 2019) recommendation systems for which the industry standard has been recently switching to contextual bandits (CB, Langford & Zhang, 2007), a simplified reinforcement learning…”
- Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement (2016, background): “…(also called estimation of treatment effects and off-policy evaluation) and counterfactual learning (also called policy optimization) has made rapid progress in recent years [8, 3, 7, 6, 2, 13, 22, 9, 11, 14, 15, 19, 24, 12, 18], and it is starting to have impact in commercial applications.”
- Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “For example, the feedback may be explicit star ratings in a movie recommendation system, and the propensities may be the results of self-selection by the users as in [21].”
- Causal Intervention for Leveraging Popularity Bias in Recommendation (2021, methodology): “Existing work on popularity bias-aware recommendation mainly performs unbiased learning or ranking adjustment, which can be categorized into: • Inverse Propensity Scoring (IPS), which adjusts the data distribution to be even by reweighting the interaction examples for model training [17, 38].”
- A Survey on Causal Inference (2020, methodology): “Based on the unbiased quality measurement, in [118], the propensity-score empirical risk minimization (ERM) for recommendation is proposed: Ŷ ∈ H is selected to optimize the following problem: Ŷ ERM = argminŶ ∈H{R̂IPS(Ŷ |P)}, where R̂IPS(Ŷ |P) is defined in Eqn.”
- Debiasing Pre-Trained Language Models via Efficient Fine-Tuning (2022, background): “Bias Issues in Machine Learning Unfair behaviors have been found in many machine learning and artificial intelligence applications, including facial recognition (Raji and Buolamwini, 2019), recommendation systems (Schnabel et al., 2016), and speech recognition (Koenecke et al., 2020).”
- Disentangling User Interest and Conformity for Recommendation with Causal Embedding (2021, background): “• IPS [26, 40]: IPS eliminates popularity bias by re-weighting each instance according to item popularity.”