High Confidence Policy Improvement

By Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh. 2015

200 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Works on high confidence policy improvement [57] provides safety guarantees for improvement but tend to be conservative.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Unlike many previous approaches to off-policy policy evaluation (Peshkin & Shelton, 2002; Thomas et al., 2015; Liu et al., 2018), our work focuses on constraining the policy to a subset of policies which can be adequately evaluated, rather than the process of evaluation itself.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “In practice, however, strict CIs are usually too pessimistic, and normal approximations are used instead (Thomas et al., 2015b).”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “…Dudı́k et al. 2011; Swaminathan et al. 2017) and reinforcement learning (RL) (e.g., Precup et al. 2000a; 2001; Paduraru 2013; Mahmood et al. 2014; Thomas et al. 2015a; Li et al. 2015; Jiang & Li 2016; Thomas & Brunskill 2016), and some of the results have been applied to problems in marketing…”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “In addition to the SPIBB algorithms, our ﬁnite MDP benchmark contains four algorithms: Basic RL, HCPI (Thomas et al., 2015a), Robust MDP, and RaMDP (Petrik et al., 2016).”
Consistent On-Line Off-Policy Evaluation (2017, background): “While this solution is impractical in problems with large state spaces parameterizing or approximating these variances (similarly to Thomas et al. (2015b)) could improve performance in specific applications.”
Safe Policy Improvement by Minimizing Robust Baseline Regret (2016, methodology): “Another class of model-free algorithms are those that use a batch of data generated by the current policy and return a policy that is guaranteed to perform better.”
Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation (2016, positive): “[18] demonstrated that approximate bounds obtained by bootstrapping importance sampled policy returns can improve data-efficiency by an order of magnitude over concentration inequalities.”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “The algorithms in Thomas, Theocharous, and Ghavamzadeh (2015b) rely on the assumption that the importance sampling estimate is normally distributed which is false when the number of trajectories is small.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[57] P.”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, background): “…sampling techniques to correct for distribution mismatch in the trajectory space, pioneered by the early work on eligibility traces (Precup et al., 2000), and further enhanced with a variety of variance reduction techniques (Thomas et al., 2015; Jiang and Li, 2016; Thomas and Brunskill, 2016).”
AI Safety Gridworlds (2017, background): “The side constraint could also be directly included in the policy optimization algorithm (Thomas et al., 2015; Achiam et al., 2017).”
Hindsight policy gradients (2017, methodology): “In reinforcement learning, importance sampling has been traditionally employed in order to efficiently reuse information obtained by earlier policies during learning (Precup et al., 2000; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Thomas et al., 2015; Munos et al., 2016).”
Offline Policy Selection under Uncertainty (2020, methodology): “…there is also a growing body of literature that uses frequentist principles to derive high-conﬁdence lower bounds for the value of a policy (Bottou et al., 2013; Thomas et al., 2015b; Hanna et al., 2016; Kuzborskij et al., 2020; Feng et al., 2020; Dai et al., 2020; Kostrikov and Nachum, 2020).”
Doubly Robust Off-policy Evaluation for Reinforcement Learning (2015, methodology): “Therefore, we select the policy that has the highest lower confidence bound using Student's t-test approximation (Thomas et al., 2015b), 6 and hold on to the current behavior policy if none of the lower confidence bounds is better than the threshold.”
Policy Optimization via Importance Sampling (2018, methodology): “Finding a suitable concentration inequality for off-policy learning was studied in [55] for offline policy evaluation and subsequently in [54] for optimization.”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, methodology): “For both the domains, (a) we set πsafe to a near-optimal policy for the starting MDP M1, representing how a doctor would have set the treatment initially, or how an expert would have set the recommendations, (b) we set the safety level (1− α) to 95%, (c) we modulate the “speed” of non-stationarity, ”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, methodology): “Again, SPI [Thomas et al., 2015b] (extended to allow for more general behavioral constraints) is an example of how Seldonian and quasi-Seldonian algorithms might be designed without this assumption (it partitions the training data as described above).”