Predictive Off-Policy Policy Evaluation for Nonstationary Decision Problems, with Applications to Digital Marketing

By Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, Emma Brunskill. 2017

56 citations recorded by the source corpus.

In this paper we consider the problem of evaluating one digital marketing policy (or more generally, a policy for an MDP with unknown transition and reward functions) using data collected from the execution of a different policy. We call this problem off-policy policy evaluation. Existing methods for off-policy policy evaluation assume that the transition and reward functions of the MDP are stationary — an assumption that is typically false, particularly for digital marketing applications. This means that existing off-policy policy evaluation methods are reactive to nonstationarity, in that they slowly correct for changes after they occur. We argue that off-policy policy evaluation for nonstationary MDPs can be phrased as a time series prediction problem, which results in predictive methods that can anticipate changes before they happen. We therefore propose a synthesis of existing off-policy policy evaluation methods with existing time series prediction methods, which we show results in a drastic reduction of mean squared error when evaluating policies using real digital marketing data set.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Nan Jiang
4. George Tucker
5. Lihong Li
6. Aviral Kumar
7. Emma Brunskill
8. Justin Fu

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…offline data has included studies on optimizing newspaper article click-through-rates (Strehl et al., 2010; Garcin et al., 2014), advertisement ranking on search pages (Bottou et al., 2013), and personalized ad recommendation for digital marketing (Theocharous et al., 2015; Thomas et al., 2017).”
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “This problem is widely encountered in many real-life applications (e.g., Murphy et al., 2001; Li et al., 2011; Bottou et al., 2013; Thomas et al., 2017), where online experiments are expensive and high-quality simulators are difficult to build.”
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “…been deployed, e.g., targeted advertising and marketing [Bottou et al., 2013; Tang et al., 2013; Chapelle et al., 2015; Theocharous et al., 2015; Thomas et al., 2017] or medical treatments [Murphy et al., 2001; Ernst et al., 2006; Raghu et al., 2017], online policy evaluation is usually…”
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations [31, 57, 3, 1], and the increasing importance of off-policy learning and counterfactual reasoning [12, 52, 37, 30, 34, 41].”
• Harnessing Infinite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, background): “This problem is widely encountered in many real-life applications (e.g., Murphy et al., 2001; Li et al., 2011; Bottou et al., 2013; Thomas et al., 2017), where online experiments are expensive and high-quality simulators are difficult to build.”
• Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “Similar assumptions have been considered for analyzing trends for digital marketing [60] and remain popular among policymakers for designing policies based on forecasting [64].”
• Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “This paradigm is useful in a diverse array of application domains such as healthcare [51, 15, 35], autonomous driving [57, 4, 36], and recommendation systems [47, 12, 49].”
• Optimizing for the Future in Non-Stationary MDPs (2020, background): “Thomas et al. (2017) and Jagerman et al. (2019) have proposed policy evaluation techniques in a non-stationary setting by tracking a policy’s past performances.”