Safe Policy Improvement by Minimizing Robust Baseline Regret

By Mohammad Ghavamzadeh, Marek Petrik, Yinlam Chow. 2016

143 citations recorded by the source corpus.

An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, i.e., a policy that is guaranteed to perform at least as well as a given baseline strategy. In this paper, we develop and analyze a new model-based approach to compute a safe policy when we have access to an inaccurate dynamics model of the system with known accuracy guarantees. Our proposed robust method uses this (inaccurate) model to directly minimize the (negative) regret w.r.t. the baseline policy. Contrary to the existing approaches, minimizing the regret allows one to improve the baseline policy in states with accurate dynamics and seamlessly fall back to the baseline policy, otherwise. We show that our formulation is NP-hard and propose an approximate algorithm. Our empirical results on several domains show that even this relatively simple approximate algorithm can significantly outperform standard approaches.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Alternatively, we can search over policies in a model of the MDP and bound the error of the estimated model with high probability (Ghavamzadeh et al., 2016; Laroche et al., 2017; Nadjahi et al., 2019).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “6) are related to prior work on safe policy improvement [33, 50], and a direct comparison to Theorems 1 and 2 in Laroche et al.”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “Other SPI algorithms use as reference the baseline performance, which is assumed to be known instead (Thomas, 2015; Petrik et al., 2016).”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “From Theorem 8 of Petrik et al. (2016), it is direct to guarantee that, if all the state-action pair counts satisfy:”
Safe Policy Improvement with an Estimated Baseline Policy (2019, background): “[21] show that this policy is approximately optimal to the order of the inverse square root of the minimal state-action pairs count in the dataset.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “Formally, Proposition 4.3 discuss safe improvement guarantees over the behavior policy. building on prior work [46, 31, 29].”
AI Safety Gridworlds (2017, background): “formation, for example through imitation learning (Abbeel et al., 2010; Santara et al., 2017): the agent could try to “stay close” to the state space covered by demonstrations pro11 vided by a human (Ghavamzadeh et al., 2016). The side constraint could also be directly included in the policy optimiz”
Confounding-Robust Policy Improvement (2018, background): “…idea of safe policy improvement by forming an uncertainty set around the presumed unknown transition probabilities between states as in Thomas et al. (2015) or forming a trust region for safe policy exploration via concentration inequalities on estimates of policy risk as in Petrik et al. (2016).”
The Importance of Pessimism in Fixed-Dataset Policy Optimization (2020, methodology): “Similar algorithms have been explored in the safe RL literature [8, 21] and the robust MDP literature [9], where algorithms with high-probability performance guarantees are useful in the context of ensuring safety.”
Bayesian Robust Optimization for Imitation Learning (2020, background): “This work has been extended to risk-averse optimization in Markov decision processes [15, 43, 44] and in the context of reinforcement learning [23, 58, 59], where the transition dynamics and reward function are not known.”
Multi-batch Reinforcement Learning (2019, background): “The literature on single batch RL focuses on safe policy improvement of the baseline policy that was used to generate the batch [6].”
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (2019, methodology): “Following analogous derivations of Lemma 11 in Petrik et al. (2016) (for the case of finite-horizon MDPs), for the case of finite-horizon MDPs, one has the following chain of inequalities for any given control sequence {ut} t=0 and initial observation x0:”
Preventing undesirable behavior of intelligent machines (2019, background): “[94] discuss a similar setting, wherein improvements must be ensured, but the algorithm is free to fall back to an existing prior solution.”
Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “Safety in RL (Garcı́a & Fernández, 2015) is an overloaded term, as it may be considered with respect to parametric uncertainty (Thomas, 2015; Petrik et al., 2016; Laroche et al., 2019), internal uncertainty (Altman, 1999; Carrara et al., 2019), or interruptibility (Orseau & Armstrong, 2016;…”
Safe Policy Improvement with Baseline Bootstrapping in Factored Environments (2019, background): “Finally, we believe it is also possible to extend other model-based safe RL meth-ods to factored environments, such as the robust approach by Petrik, Ghavamzadeh, and Chow (2016).”