Safe Exploration of State and Action Spaces in Reinforcement Learning

By Javier García, Fernando Fernández. 2012

173 citations recorded by the source corpus.

In this paper, we consider the important problem of safe exploration in reinforcement learning. While reinforcement learning is well-suited to domains with complex transition dynamics and high-dimensional state-action spaces, an additional challenge is posed by the need for safe and efficient exploration. Traditional exploration techniques are not particularly useful for solving dangerous tasks, where the trial and error process may lead to the selection of actions whose execution in some states may result in damage to the learning system (or any other system). Consequently, when an agent begins an interaction with a dangerous and high-dimensional state-action space, an important question arises; namely, that of how to avoid (or at least minimize) damage caused by the exploration of the state-action space. We introduce the PI-SRL algorithm which safely improves suboptimal albeit robust behaviors for continuous state and action control tasks and which efficiently learns from the experience gained from the environment. We evaluate the proposed method in four complex tasks: automatic car parking, pole-balancing, helicopter hovering, and business management.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Safe Model-based Reinforcement Learning with Stability Guarantees (2017, background): “In [20, 21], a priori known, safe global backup policies are used, while [22] learns to switch between several safe policies.”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, methodology): “For example, the work in Berkenkamp et al. [2017] and Garcia and Fern´andez [2012] approaches the problem of safe exploration, the former by using a Lyapunov stability veriﬁcation and the latter by gradually improving over a predeﬁned (and safe) baseline policy (For a comprehensive review of…”
Counterfactual Risk Minimization (2015, methodology): “The training objectiveis decomposed using repeated variance linearization, and optimizing it using stochastic gradient descent yields a fast and effective algorithm.”
Batch learning from logged bandit feedback through counterfactual risk minimization (2015, background): “…mimics similar successful approaches in other problems like supervised learning (Wapnik and Tscherwonenkis, 1979), risk averse multi-armed bandits (Galichet et al., 2013), regret minimizing contextual bandits (Langford and Zhang, 2008) and reinforcement learning (Garcia and Fernandez, 2012).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Some methods are proposed to deal with this problem, such as model-based safe RL to plan safe actions [272], Lyapunov function to guarantee safety of agents, predefined baseline policy for safe exploration [90], formal verification for safe autonomous driving [189], constrained Q learning for high-l”
Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning (2017, background): “In safe RL numerous other approaches exists for safe exploration (Joseph et al., 2013; Hans et al., 2008; Garcia & Fernández, 2012; Berkenkamp et al., 2017).”
Safe Exploration for Optimization with Gaussian Processes (2015, background): “These problems model sequential decision tasks, in which one chooses among a number of different decisions (arms), each associated with a stochastic reward with initially unknown distribution.”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “Garcia and Fernandez [134] baseline behaviour Human Designer Knowledge car parking problem 6.”
A Survey of State-Action Representations for Autonomous Driving (2018, methodology): “…is composed of the longitudinal acceleration and steering angle, and often used for low-level control tasks, such as (Sadigh et al., 2016; Cardamone et al., 2009; Ross et al., 2011; Levine and Koltun, 2012; Garcia and Fernandez, 2012; Koutn(cid:237)k et al., 2013; Van Den Heuvel et al., 2013).”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, methodology): “In Section 5, we apply action-pruning to prevent catastrophic outcomes during exploration, a problem explored by Lipton et al. [27], Garcia and Fernandez [14, 15], Hans et al. [18], Moldovan and Abbeel [31].”
Safe Exploration in Finite Markov Decision Processes with Gaussian Processes (2016, background): “Garcia and Fernández [6] propose to ensure safety by means of a backup policy; that is, a policy that is known to be safe in advance.”
Artificial intelligence, bias and clinical safety (2019, unknown): “How do we set limits to prevent dangerous overdosing, and define what changes in strategy are safe for the system to ‘explore’ 50 ?”
Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation (2019, background): “Garcia and Fernández [9] perturb a baseline policy and request action feedback if the current state exceeds a minimum distance from any demonstration.”
A comprehensive survey on safe reinforcement learning (2015, methodology): “However, in some situations in which the safety of the agent is particularly important, for example in expensive robotic platforms, researchers are paying increasing attention not only to the long-term reward maximization, but also to damage avoidance (Mihatsch and Neuneier, 2002; Hans et al., 2008;”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “In addition, the asymptotic results that safe RL methods can achieve are summarised in the preprint version of the paper [76].”