Safe Exploration in Finite Markov Decision Processes with Gaussian Processes

By Matteo Turchetta, Felix Berkenkamp, Andreas Krause. 2016

204 citations recorded by the source corpus.

In classical reinforcement learning, when exploring an environment, agents accept arbitrary short term loss for long term gain. This is infeasible for safety critical applications, such as robotics, where even a single unsafe action may cause system failure. In this paper, we address the problem of safely exploring finite Markov decision processes (MDP). We define safety in terms of an, a priori unknown, safety constraint that depends on states and actions. We aim to explore the MDP under this constraint, assuming that the unknown function satisfies regularity conditions expressed via a Gaussian process prior. We develop a novel algorithm for this task and prove that it is able to completely explore the safely reachable part of the MDP without violating the safety constraint. To achieve this, it cautiously explores safe states and actions in order to gain statistical confidence about the safety of unvisited state-action pairs from noisy observations collected while navigating the environment. Moreover, the algorithm explicitly considers reachability when exploring the MDP, ensuring that it does not get stuck in any state with no safe way out. We demonstrate our method on digital terrain models for the task of exploring an unknown map with a rover.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, methodology): “Finally, we thank the Google Brain team for providing a supportive environment and encouraging us to publish this work.”
Verifiable Reinforcement Learning via Policy Extraction (2018, background): “Specific to reinforcement learning, there has been substantial interest in safe exploration [23, 36, 33]; see [14] for a survey.”
Safe Model-based Reinforcement Learning with Stability Guarantees (2017, background): “Both [11] and [12] introduce algorithms to safely explore MDPs so that the agent never gets stuck without safe actions.”
Challenges of Real-World Reinforcement Learning (2019, methodology): “Other RL approaches include using Lyapunov functions to learn safe policies (Chow et al., 2018) and safe exploration strategies that predict the safety of neighboring states (Turchetta et al., 2016; Wachi et al., 2018).”
AI Safety Gridworlds (2017, background): “There are several possible formulations for safe exploration: being able to return to the starting state or some other safe state with high probability (Moldovan and Abbeel, 2012), achieving a minimum reward (Hans et al., 2008), or satisfying a given side constraint (Turchetta et al., 2016).”
A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems (2017, background): “Later safe exploration methods have assumed fully known deterministic dynamics with uncertain the safety features [46], or allowed for uncertain dynamics but restricted the class of constraints to local stability (i.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Based on a Gaussian process of a safe RL method [285], SNO-MDP [284] is developed to optimise cost in the safe region and optimise the reward in the unknown safety constrained region [260, 280].”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, background): “Prior work has studied safety in RL in several ways, including imposing constraints on expected return [1, 34], risk measures [17, 29, 31, 33], and avoiding regions of the MDP where constraint violations are likely [4, 5, 10, 12, 36, 37].”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “1 Nonlinear (69, 79, 82), GP (78, 79, 80, 81) NN (70, 75, 85, 86, 87), Tabular (73) Input & state (69, 70, 73, 75, 78, 79, 80, 81, 82) Trajectory (87) Input & state (69, 70, 73, 75, 78, 79, 80, 81, 82) Trajectory (85, 86, 87) Stabilization (81), Tracking (79), Navigation (69, 70, 73, 82, 85, 86), Lo”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “Besides, based on a Gaussian process of a safe RL method [46], the SNO-MDP 6 [141] is developed to optimize cost in the safe region and optimize the reward in the unknown safety-constrained region [44], [45].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “2018) and exploration strategies that predict the safety of neighboring states (Turchetta et al. 2016; Wachi et al. 2018).”
Measuring and avoiding side effects using relative reachability (2018, background): “Safe exploration methods prevent the agent from taking harmful actions by enforcing safety constraints (Turchetta et al., 2016; Dalal et al., 2018), using intrinsic motivation (Lipton et al., 2016), penalizing risk (Chow et al., 2015; Mihatsch and Neuneier, 2002), preserving ergodicity (Moldovan…”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “Safe exploration methods prevent the agent from taking harmful actions by enforcing safety constraints [Turchetta et al., 2016], penalizing risk [Chow et al., 2015], using intrinsic motivation [Lipton et al., 2016], preserving reversibility [Moldovan and Abbeel, 2012; Eysenbach et al., 2017], etc.…”
Transforming task representations to perform novel tasks (2020, background): “It could also be useful in settings like robotics, where mistakes during learning can be extremely costly (28).”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “This can be either in the form of a mathematical model [139] or heuristic rules [140].”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “2018] and exploration strategies that predict the safety of neighboring states [Turchetta et al. 2016; Wachi et al. 2018].”
Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics (2016, unknown): “…stable for all possible models within the uncertainty speciﬁcation (Berkenkamp and Schoellig 2015; Berkenkamp et al. 2017) or the system never leaves a safe subset of the state space (Ostafew et al. 2016; Aswani et al. 2013; Akametalu et al. 2014; Moldovan and Abbeel 2012; Turchetta et al. 2016).”