A Lyapunov-based Approach to Safe Reinforcement Learning

By Yinlam Chow, Ofir Nachum, Edgar A. Duéñez-Guzmán, Mohammad Ghavamzadeh. Neural Information Processing Systems, 2018

665 citations recorded by the source corpus.

In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance it is crucial to guarantee the safety of an agent during training as well as deployment (e.g. a robot should avoid taking actions - exploratory or not - which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision problems (CMDPs), an extension of the standard Markov decision problems (MDPs) augmented with constraints on expected cumulative costs. Our approach hinges on a novel \emph{Lyapunov} method. We define and present a method for constructing Lyapunov functions, which provide an effective way to guarantee the global safety of a behavior policy during training via a set of local, linear constraints. Leveraging these theoretical underpinnings, we show how to use the Lyapunov approach to systematically transform dynamic programming (DP) and RL algorithms into their safe counterparts. To illustrate their effectiveness, we evaluate these algorithms in several CMDP planning and decision-making tasks on a safety benchmark domain. Our results show that our proposed method significantly outperforms existing baselines in balancing constraint satisfaction and performance.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/65fb1b37c41902793ac65db3532a6e51631a9aff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, neutral): “Fromtheperspectiveofknowledgeutilization, somemethodsconsidersafetyintheRLexplorationprocessby leveragingexternalknowledge,e.g.,[34],[100],[101],[102], [103],[104],[105].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “Additional RL approaches include using Lyapunov functions to learn safe policies (Chow et al. 2018) and exploration strategies that predict the safety of neighboring states (Tur-chetta et al. 2016; Wachi et al. 2018).”
Neural Lyapunov Control (2020, methodology): “Related learning-based approaches for finding Lyapunov functions include [6, 7, 10, 23].”
Safely Learning to Control the Constrained Linear Quadratic Regulator (2018, background): “Finally, the machine-learning community has begun to consider safety in reinforcement learning, where much work positions itself as being for general dynamical systems in lieu of providing statistical guarantees [16]–[19].”
Neurosymbolic Programming (2021, background): “Most approaches to the problem estimate the safety of policies statistically (Moldovan and Abbeel, 2012; Chow et al., 2018; Achiam et al., 2017; Chow et al., 2018).”
VALUATION : A N ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES (2018, neutral): “…these address situations where costs are observed frequently, and the objective is to keep expected costs below some threshold (Achiam et al., 2017; Chow et al., 2018) as opposed to avoiding costs entirely, and are not designed to speciﬁcally satisfy constraints when violations would be caused by…”
MuZero with Self-competition for Rate Control in VP9 Video Compression (2022, background): “J ck π ≤ βk ∀k (2) There are a number of approaches to solving this constrained RL objective [Altman, 1999, Tessler et al., 2018, Efroni et al., 2020, Achiam et al., 2017, Bohez et al., 2019, Chow et al., 2018, Paternain et al., 2019, Zhang et al., 2020].”
Safe Control With Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction Methods for Robotics and Control (2022, methodology): “Lyapunov functions from a cMDP’s cost metric [39], and later work [40] has extended this approach to continuous state and action spaces.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “In subsequent work (98), the authors of (97) extend the approach to continuous input spaces and standard policy gradient methods, addressing its computational tractability.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “control [17, 132, 139, 183, 204, 231, 258], adaptive control [89, 115, 199, 238], robust control [26, 73, 112, 282], lyapunov function [56, 57] for RL stability and safety.”
A Distributional View on Multi-Objective Policy Optimization (2020, background): “Similarly, safe RL is concerned with learning policies that optimize a scalar reward while not violating safety constraints (Achiam et al., 2017; Chow et al., 2018); this has also been studied in the off-policy batch RL setting (Le et al.”
End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks (2019, methodology): “…methods or model predictive control to guarantee safety under system dynamics during learning (Wang, Theodorou, and Egerstedt 2017; Berkenkamp et al. 2017; Chow et al. 2018; Ohnishi et al. 2018; Koller et al. 2018), but they do not address the issue of exploration and performance optimization.”
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, methodology): “Approaches to safe RL based on Lyapunov functions have been developed in a series of studies (Chow et al., 2018; 2019), resulting in algorithms that combine a projection step, as in CPO, with action-layer interventions like the safety layer of Dalal et al.”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “Additional RL approaches include using Lyapunov functions to learn safe policies [Chow et al. 2018] and exploration strategies that predict the safety of neighboring states [Turchetta et al. 2016; Wachi et al. 2018].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “While there are abundant works on safety for RL (Chow et al., 2015; Eysenbach et al., 2018; Chow et al., 2018; Ray et al., 2019; Chow et al., 2019), they often do not provide guarantees in practice when combined with neural networks and stochastic optimization.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “For ﬁnite-horizon non-discounted MDPs, the value V or action-conditional value Q of a policy π is which may be extended to regularize the policy’s entropy with some temperature α ≥ 0 as The value function of a given policy can be estimated ( i.e. policy evaluation ) by explicitly rolling out the wor”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, neutral): “Conventionally, RL algorithms designed to ensure safety [47, 26, 58, 69, 38, 17] model the environment as a Markov decision process (MDP), and rely upon the stationarity assumption made by MDPs [53].”
Learning Stable Deep Dynamics Models (2020, background): “Lyapunov functions in deep learning Finally, there has been a small set of recent work exploring the intersection of deep learning and Lyapunov analysis [5, 17, 19].”
CAQL: Continuous Action Q-Learning (2019, background): “Furthermore, the admissible action set in RL is constrained in practice, for example, when actions must lie within a specific range for safety (Chow et al., 2018).”
Model-augmented Prioritized Experience Replay (2022, methodology): “For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed in parallel of methods for improving the sample efficiency of RL.”