Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

By Adam Stooke, Joshua Achiam, Pieter Abbeel. International Conference on Machine Learning, 2020

463 citations recorded by the source corpus.

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as \emph{integral} control; our terms introduce \emph{proportional} and \emph{derivative} control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

Subfields: Computational Theory and Mathematics, Control and Systems Engineering, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/629d0ce250581471f07083bbab95f23623b00201

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Control and Systems Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Stooke et al. (2020) introduce the concept of lagrangian damping which leads to improved stability by performing PID control on the lagrangian parameter.”