Off-policy Bandits with Deficient Support
By Noveen Sachdeva, Yi-Hsun Su, Thorsten Joachims. 2020
88 citations recorded by the source corpus.
Learning effective contextual-bandit policies from past actions of a deployed system is highly desirable in many settings (e.g. voice assistants, recommendation, search), since it enables the reuse of large amounts of log data. State-of-the-art methods for such off-policy learning, however, are based on inverse propensity score (IPS) weighting. A key theoretical requirement of IPS weighting is that the policy that logged the data has "full support", which typically translates into requiring non-zero probability for any action in any context. Unfortunately, many real-world systems produce support deficient data, especially when the action space is large, and we show how existing methods can fail catastrophically. To overcome this gap between theory and applications, we identify three approaches that provide various guarantees for IPS-based learning despite the inherent limitations of support-deficient data: restricting the action space, reward extrapolation, and restricting the policy space. We systematically analyze the statistical and computational properties of these three approaches, and we empirically evaluate their effectiveness. In addition to providing the first systematic analysis of support-deficiency in contextual-bandit learning, we conclude with recommendations that provide practical guidance.
Subfields: Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Sergey Levine
- Aviral Kumar
- George Tucker
- Justin Fu
- Ofir Nachum
- Shixiang Shane Gu
- Chelsea Finn
- Pieter Abbeel
Citations in the corpus
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “(21) This metric has been used in the context of off-policy bandits (Sachdeva et al., 2020), but not in the context of offline reinforcement learning.”
- SVP-CF: Selection via Proxy for Collaborative Filtering Data (2021, background): “A wide variety of ways exist to model the propensity-score of a user-item interaction (Schnabel & Bennett, 2020; Jain et al., 2016; Sachdeva et al., 2020).”