Safe Model-based Reinforcement Learning with Stability Guarantees

By Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, Andreas Krause. 2017

1,021 citations recorded by the source corpus.

Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-critical systems in the real world. In this paper, we present a learning algorithm that explicitly considers safety, defined in terms of stability guarantees. Specifically, we extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Moreover, under additional regularity assumptions in terms of a Gaussian process prior, we prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space. In our experiments, we show how the resulting algorithm can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Methods that guarantee Lyapunov stability (Berkenkamp et al., 2017) of the learned model can be used to constrain policies within “safe” regions of the state space where the chance of failure is small.”
Uncertainty in Multitask Transfer Learning (2018, background): “This is indeed the case for Bayesian optimization [30], Bayesian active learning [12], continual learning [20], safe reinforcement learning [4], exploration-exploitation trade-off in reinforcement learning [17].”
Safe Policy Improvement with Baseline Bootstrapping (2017, background): “(Berkenkamp et al., 2017) assume the existence of a local, stable policy, and their Safe Lyapunov RL algorithm allows to safely explore outside from the safe region without ever leaving it.”
End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks (2019, methodology): “…Lyapunov-based methods or model predictive control to guarantee safety under system dynamics during learning (Wang, Theodorou, and Egerstedt 2017; Berkenkamp et al. 2017; Chow et al. 2018; Ohnishi et al. 2018; Koller et al. 2018), but they do not address the issue of exploration and performance…”
Neural Lyapunov Control (2020, methodology): “Related learning-based approaches for finding Lyapunov functions include [6, 7, 10, 23].”
Safe Reinforcement Learning Using Robust MPC (2019, background): “Thecombination of learning and control techniques has been proposed in, e.g., [13], [14], [15], [16], [17].”
Data-Driven Model Predictive Control With Stability and Robustness Guarantees (2019, methodology): “The main advantage of the proposed MPC scheme over existing adaptive or learning-based methods such as [5], [6], [7], [8], [9] is that it requires only an initially measured, persistently exciting data trajectory as well as an upper bound on the system order, but no (set-based) model description and”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, neutral): “Some attempts [165]–[168] propose safe on-policy exploration schemes based on Lyapunov criterion and Gaussian process.”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, background): “Therefore, in order to ultimately establish a truly safe and trustworthy model, further steps are needed, potentially including the use of formal veriﬁcation methods [19], [66].”
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks (2019, background): “A central problem in learningbased control is to provide stability or safety guarantees for a feedback control loop when a learningenabled component, such as a deep neural network, is introduced in the loop [4, 8, 20].”
How RL Agents Behave When Their Actions Are Modified (2021, neutral): “For example, action constraints have been used to prevent damage when training robot policies [8, 15] and transformations to the policy can be used to ensure that an agent stays within some safe region of state space [3, 5].”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “This setting can likely be addressed by leveraging ideas from the literature in safe reinforcement learning [11, 42, 3, 5].”
Learning-Based Model Predictive Control: Toward Safe Learning in Control (2020, background): “The data-driven computation of such safe sets together with safety controllers has been studied for different classes of systems (see, e.g., 132–135).”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, result): “thms is typically very high, which limits their use on real-world systems [14, 29]. Model-based RL, on the other hand, aims to learn a model of the system, and then optimize a policy given this model [4, 13, 18, 25]. This approach has the benefit of reducing sample complexity, but the scope of the p”
Combating the Compounding-Error Problem with a Multi-step Model (2019, methodology): “Following previous work [Berkenkamp et al. , 2017; Asadi et al. , 2018b; Luo et al. , 2018; Gelada et al. , 2019] we make assumptions on the smoothness of models, characterized below.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “Besides, safe policy improvement methods [23, 44, 6, 11] require a separately estimated model to μ to deal with unseen actions.”