End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

By Richard Cheng, Gábor Orosz, Richard M. Murray, Joel W. Burdick. AAAI Conference on Artificial Intelligence, 2019

805 citations recorded by the source corpus.

Reinforcement Learning (RL) algorithms have found limited success beyond simulated applications, and one main reason is the absence of safety guarantees during the learning process. Real world systems would realistically fail or break before an optimal controller can be learned. To address this issue, we propose a controller architecture that combines (1) a model-free RL-based controller with (2) model-based controllers utilizing control barrier functions (CBFs) and (3) online learning of the unknown system dynamics, in order to ensure safety during learning. Our general framework leverages the success of RL algorithms to learn high-performance controllers, while the CBF-based controllers both guarantee safety and guide the learning process by constraining the set of explorable polices. We utilize Gaussian Processes (GPs) to model the system dynamics and its uncertainties. Our novel controller synthesis algorithm, RL-CBF, guarantees safety with high probability during the learning process, regardless of the RL algorithm used, and demonstrates greater policy exploration efficiency. We test our algorithm on (1) control of an inverted pendulum and (2) autonomous carfollowing with wireless vehicle-to-vehicle communication, and show that our algorithm attains much greater sample efficiency in learning than other state-of-the-art algorithms and maintains safety during the entire learning process.

Subfields: Automotive Engineering, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/adcd4bfd88213da0c33d3cb7057411dd15b72c7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “…common ways to handle soft constraints include 1) using barrier functions or penalty functions to integrate the constraints to the reward function [172]; 2) modeling as a chance constraint (i.e., the probability of constraint violation is no larger than a predeﬁned threshold) [173], [174] or a…”
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020, background): “In this direction, recent works have shown promising results towards safe DRL that is able to ensure convergence even while reducing the exploration space [3].”
Model-augmented Prioritized Experience Replay (2022, methodology): “For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed in parallel of methods for improving the sample efficiency of RL.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “Input & state (123, 124, 125, 126, 127, 128, 129, 130, 139, 140, 141, 142), State (133, 134, 135) (123, 124, 125, 126, 127, 128, 129, 130, 133, 134, 135, 139, 140, 141) Input & state (119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 139, 140, 141, 142), State (133, 134, 135, 136) (11”
Safe Control With Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction Methods for Robotics and Control (2022, methodology): “The first class of methods use an a priori given certificate to constrain the actions that the RL agent can take [33], [34], [35], possibly with some learning of model uncertainty as the controller explores [33], [34].”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, positive): “Moreover,[158]leveragesamodel-free RLandCBFstoguaranteelearningsafetywhileimproving learningefﬁciency,inwhichCBFsareusedtoconstrainthe searchspacetoenhancelearningsafetyandefﬁciencybased onaGPlearningmodel.”