Offline policy evaluation across representations with applications to educational games

By Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, Zoran Popovic. 2014

209 citations recorded by the source corpus.

Consider an autonomous teacher agent trying to adaptively sequence material to best keep a student engaged, or a medical agent trying to help suggest treatments to maximize patient outcomes. To solve these complex reinforcement learning problems, we must first decide on a policy representation. But determining the best representation can be challenging, since the environment includes many poorly-understood processes (such as student engagement) and is therefore difficult to accurately simulate. These domains are also high stakes, making it infeasible to evaluate candidate representations by running them online. Instead, one must leverage existing data to learn and evaluate new policies for future use. In this paper, we present a data-driven methodology for comparing and validating policies offline. Our method is unbiased, agnostic to representation, and focuses on the ability of each policy to generalize to new data. We apply this methodology to a partially-observable, high-dimensional concept sequencing problem in an educational game. Guided by our evaluation methodology, we propose a novel feature compaction method that substantially improves policy performance on this problem. We deploy the best-performing policies to 2,000 real students and show that the learned adaptive policy shows statistically significant improvement over random and expert baselines, improving our achievement-based reward measure by 32%.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, background): “…deployed some policy, e.g., for determining which advertisement to show a user visiting a website (Theocharous et al., 2015), for determining which medical treatment to suggest for a patient (Thapa et al., 2005), or for suggesting a personalized curriculum for a student (Mandel et al., 2014).”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “On benchmark problems, the new estimator is much more accurate than importance sampling baselines, while remaining unbiased in contrast to regression-based approaches.”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, background): “Reinforcement learning (RL) in settings such as healthcare (Murphy 2003) and education (Mandel et al. 2014) is often limited to the offline or off-policy setting, where we only use existing observed data, due to the inability to simulate and the costliness of exploration.”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “, 2011), and education (Mandel et al., 2014), the deployment of a new policy, even just for the sake of performance evaluation, may be expensive and risky.”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “This problem is of great practical importance in many application domains where deploying a new policy can be costly or risky, such as medical treatments [26], econometrics [13], recommender systems [19], education [23], Web search [18], advertising and marketing [4, 5, 38, 40].”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “…et al. 2015a; Li et al. 2015; Jiang & Li 2016; Thomas & Brunskill 2016), and some of the results have been applied to problems in marketing (e.g., Li et al. 2011; Theocharous et al. 2015), healthcare (e.g., Murphy et al. 2001; Hirano et al. 2003), and education (e.g., Mandel et al. 2014; 2016).”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “High-Confidence PI refers to the family of algorithms introduced in [31], which rely on the ability to produce high-confidence lower bound on the Importance Sampling (IS) estimate of the target policy performance, which is known to be high variance [18, 34, 22].”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “…tractable and theoretically-grounded family of frequentist algorithms appeared under the name of High Con-ﬁdence Policy Improvement (Paduraru 2013; Mandel et al. 2014; Thomas, Theocharous, and Ghavamzadeh 2015a, HCPI), relying on importance sampling estimates of the trained policy performance.”
A Minimalist Approach to Offline Reinforcement Learning (2021, background): “This questions the empirical effectiveness of offline RL for safety-critical real-world use cases [Mandel et al., 2014, Gottesman et al., 2018, Gauci et al., 2018, Jaques et al., 2019, Matsushima et al., 2020] as well as the current trend of reporting only the mean-value of the final policy in…”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “Offline RL is an important area of research in reinforcement learning, as its problem setting is more suited for learning from costly real-world data, such as in robotics [16, 22], health [33], dialog agents [20], and education [29].”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “…reinforcement learning, has been studied extensively in domains such as healthcare (Thapa et al., 2005; Raghu et al., 2018), recommender systems (Dudík et al., 2014; Theocharous et al., 2015; Swaminathan et al., 2017), education (Mandel et al., 2014), and robotics (Kalashnikov et al., 2018).”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, background): “OPE is a key problem in reinforcement learning (RL) (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016) and it finds applications as varied as healthcare (Murphy, 2003) and education (Mandel et al., 2014).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “However, in many real-world applications of RL, such as health (Murphy et al., 2001), education (Mandel et al., 2014), dialog agents (Jaques et al., 2019), and robotics (Gu et al., 2017a; Kalashnikov et al., 2018), the deployment of a new data-collection policy may be associated with a number of…”
Representation Balancing MDPs for Off-Policy Policy Evaluation (2018, methodology): “A second common approach is to estimate a dynamics and reward model, which can substantially reduce variance, but can be biased and inconsistent (as noted by [15]).”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “…Jiang and Li, 2016; Thomas and Brunskill, 2016; Guo et al., 2017; Farajtabar et al., 2018] tends to be too high to provide informative results, for long-horizon problems [Mandel et al., 2014], since the variance of the product of importance weights may grow exponentially as the horizon goes long.”
Offline Evaluation of Online Reinforcement Learning Algorithms (2016, background): “These problems have ignited a recent renewal of interest in ofﬂine policy evaluation in the RL community (Mandel et al. 2014; Thomas, Theocharous, and Ghavamzadeh 2015), where one uses a precollected dataset to achieve high-quality estimates of the performance of a proposed policy.”
Sample Efficient Policy Search for Optimal Stopping Domains (2017, result): “As has been noted by prior work [Mandel et al. , 2014] model-ﬁtting procedures focus on maximizing the likelihood of the observed data rather than trying to directly identify a policy that is expected to perform well.”