Lyapunov-based Safe Policy Optimization for Continuous Control

By Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, Edgar A. Duéñez-Guzmán. arXiv.org, 2019

297 citations recorded by the source corpus.

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction. Videos of the experiments can be found in the following link: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3fa50569925cfecc66fed5ec616682ecf3794ad7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “The work in (99) proposes Backward Value Functions (BVF) as a way to overcome the excessive computational cost of the approaches in the previous paragraphs (34, 98).”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Many researchers have proposed extensions of the existing deep RL algorithms to address safety, such as learning an additional safety layer that projects raw actions to a safe feasibility set [3, 25, 20, 17], learning the boundary of the safe states with a classifier [38, 49, 23], expanding the iden”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Unlike [214], in [56] and [57], Chow et al.”
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives (2023, methodology): “[94] propose safe policy optimization algorithms that employ a Lyapunov-based approach [95] to address CMDP problems.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “Reinforcement learning (RL) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-making tasks (Barth-Maron et al., 2018; Hessel et al., 2018; Nachum et al., 2019).”
Projection-Based Constrained Policy Optimization (2020, unknown): “Similarly, Chow et al. (2019) proposes θ -projection.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “Unlike [66], in [34] and [35] by a proposed Linear Programming algorithm.”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, methodology): “…algorithms for safe RL involves modifying the policy optimization procedure of standard RL algorithms to simultaneously reason about both task reward and constraints using methods such as trust regions [1], optimizing a Lagrangian relaxation [25, 30, 34], or constructing Lyapunov functions [6, 7].”
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, methodology): “λ0 = 1 becomes an obvious choice for initialization, a point where previous theory provides little guidance (Chow et al., 2019) (although here we left λ0 = 0).”