Safe Exploration in Continuous Action Spaces

By Gal Dalal, Krishnamurthy (Dj) Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, Yuval Tassa. arXiv.org, 2018

522 citations recorded by the source corpus.

We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be violated. We show how to exploit the typically smooth dynamics of these systems and enable RL algorithms to never violate constraints during learning. Our technique is to directly add to the policy a safety layer that analytically solves an action correction formulation per each state. The novelty of obtaining an elegant closed-form solution is attained due to a linearized model, learned on past trajectories consisting of arbitrary actions. This is to mimic the real-world circumstances where data logs were generated with a behavior policy that is implausible to describe mathematically; such cases render the known safety-aware off-policy methods inapplicable. We demonstrate the efficacy of our approach on new representative physics-based environments, and prevail where reward shaping fails by maintaining zero constraint violations.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7f567df97dc7e099d96e6c590ddf5aef8c5b11c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Reward Constrained Policy Optimization (2018, background): “Discounted sum approaches such as Achiam et al. (2017) and per-state constraints such as Dalal et al. (2018) are unsuitable for comparison given the considered constraints.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recent work in RL safety (Dalal et al. 2018; Achiam et al. 2017; Tessler et al. 2018; Satija et al. 2020) has cast safety in the context of Constrained MDPs (CMDPs) (Altman 1999), and we will concentrate on pre-defined constraints on the environment in this context.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “Rather than designing for a recoverable exploration through ergodicity, safe exploration strategies that provide constraint satisfaction are developed in (75, 70).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “roposed set of constraints are physically bound and are intended to avoid self-harm, but the suite’s framework provides options for users to deﬁne any constraints they wish. Recent work in RL safety [Dalal et al. 2018; Achiam et al. 2017] has cast safety in the context of Constrained MDPs (CMDPs) [A”
How RL Agents Behave When Their Actions Are Modified (2021, unknown): “For example, action constraints have been used to prevent damage when training robot policies [8, 15] and transformations to the policy can be used to ensure that an agent stays within some safe region of state space [3, 5].”
Learning Human Objectives by Evaluating Hypothetical Behavior (2019, background): “ReQueST is also complementary to prior work on safe exploration, which typically assumes a known reward function and side con-Learning straints, and focuses on ensuring that the agent never visits unsafe states during policy optimization (Dalal et al., 2018; Garcıa & Fern´andez, 2015).”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Many researchers have proposed extensions of the existing deep RL algorithms to address safety, such as learning an additional safety layer that projects raw actions to a safe feasibility set [3, 25, 20, 17], learning the boundary of the safe states with a classifier [38, 49, 23], expanding the iden”
Batch Policy Learning under Constraints (2019, background): “Other approaches (Cheng et al., 2019; Dalal et al., 2018) address safe exploration by building the constraint directly into the policy.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “For ﬁnite-horizon non-discounted MDPs, the value V or action-conditional value Q of a policy π is which may be extended to regularize the policy’s entropy with some temperature α ≥ 0 as The value function of a given policy can be estimated ( i.e. policy evaluation ) by explicitly rolling out the wor”
Adaptive Power System Emergency Control Using Deep Reinforcement Learning (2019, background): “Recently, constrained policy optimization [42] and safe exploration [43]”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “Finally, in [29], Fulton and Platzer also use external knowledge to ensure agent safety by leveraging the justiﬁed speculative control sandbox in ofﬂine formal veriﬁcation settings.”
Measuring and avoiding side effects using relative reachability (2018, background): “Safe exploration methods prevent the agent from taking harmful actions by enforcing safety constraints (Turchetta et al., 2016; Dalal et al., 2018), using intrinsic motivation (Lipton et al., 2016), penalizing risk (Chow et al., 2015; Mihatsch and Neuneier, 2002), preserving ergodicity (Moldovan…”
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, methodology): “Approaches to safe RL based on Lyapunov functions have been developed in a series of studies (Chow et al., 2018; 2019), resulting in algorithms that combine a projection step, as in CPO, with action-layer interventions like the safety layer of Dalal et al. (2018).”
Let's Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety (2019, background): “[Dalal et al. 2018] Dalal, G.; Dvijotham, K.; Vecerik, M.; Hester, T.; Paduraru, C.; and Tassa, Y. 2018.”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, background): “Integrating prior knowledge about the system transitions is exploited in order to project the action chosen by the policy to a set that ensures the satisfaction of the constraints [21].”
Challenges of Real-World Reinforcement Learning (2019, methodology): “Accumulating all these violations into a single number has been proposed previously (Dalal et al., 2018) and provides a good global summary.”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, methodology): “Our second approach, which we call action projection or a -projection , uses the concept of a safety layer introduced by Dalal et al. (2018) to handle simple single-step constraints, extends this concept to general trajectory-based constraints, solves the constrained policy optimization problem in…”