Top-K Off-Policy Correction for a REINFORCE Recommender System

By Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, Ed H. Chi. 2018

558 citations recorded by the source corpus.

Industrial recommender systems deal with extremely large action spaces -- many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (e.g., user clicks, dwell time) are available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at Youtube, built with a policy-gradient-based algorithm, i.e. REINFORCE. The contributions of the paper are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions; (2) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our approaches through a series of simulations and multiple live experiments on Youtube.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

Citations in the corpus

MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
Horizon: Facebook's Open Source Applied Reinforcement Learning Platform (2018, background): “is high (Chen et al., 2019; Zhao et al., 2018b;a; Mirhoseini et al., 2017; Zheng et al., 2018), the current set of implementations and tooling must be adapted to handle the unique challenges faced in applied settings.”
Batch Reinforcement Learning Through Continuation Method (2021, background): “While RL is fundamentally an online learning paradigm, many practical applications of RL algorithms, e.g., recommender systems [5, 7] or autonomous driving [36], fall under the batch RL setup.”
Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback (2019, methodology): “However, propensitybased estimators often suffer from a high variance [4, 22, 25].”
Bias and Debias in Recommender System: A Survey and Future Directions (2020, background): “Inspired by [157], some work [156], [158], [159] further explore off-policy evaluation for non-stationary recommendation environments or slate recommendation.”
Seq2Slate: Re-ranking and Slate Optimization with RNNs (2018, background): “Finally, an interesting future direction is to study off-policy correction for seq2slate [Joachims et al., 2018, Chen et al., 2019].”
Optimizing Long-term Social Welfare in Recommender Systems: A Constrained Matching Approach (2020, background): “Some work has looked at the impact of RS policies on provider welfare (Singh & Joachims, 2018), on long-term user and provider metrics (e.g., using RL (Chen et al., 2018; Ie et al., 2019b) or assessing various other phenomena (Ribeiro et al., 2020; Celma, 2010)).”
Reinforcement Learning based Recommender Systems: A Survey (2021, methodology): “[25] adapts REINFORCE algorithm to a neural candidate generator with a very large action space.”
Fairness in Recommendation Ranking through Pairwise Comparisons (2019, methodology): “Two that relate to the challenges in fairness are the temporal dynamics [33, 48, 26, 9] and biased training data [29, 15, 3].”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “…our setting is completely oﬀ-policy with no opportunity of collecting new data from arbitrary policies, as considered in, e.g. , Athey and Wager (2017); Kallus (2018); Kallus and Zhou (2018); Swaminathan and Joachims (2015) for H = 0 and Chen et al. (2019); Fujimoto et al. (2019) for H ≥ 1.”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, background): “Interest in the application of RL to optimizing these sequences has been rarer [Shani et al., 2005] though the recent successes of deep RL have spurred activity in the use of RL for recommendation [Gauci et al., 2018, Zheng et al., 2018, Chen et al., 2018, Zhao et al., 2018, Ie et al., 2019].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recommender systems are amongst the most solicited large-scale software systems, and RL proposes an enticing framework for optimizing them (Covington et al. 2016; Chen et al. 2019a).”