Safe Reinforcement Learning via Shielding

By Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu. AAAI Conference on Artificial Intelligence, 2017

968 citations recorded by the source corpus.

Reinforcement learning algorithms discover policies that maximize reward, but do not necessarily guarantee safety during learning or execution phases. We introduce a new approach to learn optimal policies while enforcing properties expressed in temporal logic. To this end, given the temporal logic specification that is to be obeyed by the learning system, we propose to synthesize a reactive system called a shield. The shield monitors the actions from the learner and corrects them only if the chosen action causes a violation of the specification. We discuss which requirements a shield must meet to preserve the convergence guarantees of the learner. Finally, we demonstrate the versatility of our approach on several challenging reinforcement learning scenarios.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/e4b4066e46fd160cb84e246b0895a4cd674d8ce4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks (2019, background): “The concept is akin to shielded RL (Alshiekh et al. 2017; Fisac et al. 2018), since the CBF controller compensates for the RL controller to ensure safety, but does not guide exploration of the RL algorithm.”
Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation (2019, background): “Similarly, “shielding” [3] uses a manually specified safety constraint and a coarse, conservative abstraction of the dynamics to prevent an agent from violating the safety constraint.”
PAC Confidence Predictions for Deep Neural Network Classifiers (2020, background): “There has also been work on safe learning-based control (Akametalu et al., 2014; Fisac et al., 2019; Bastani, 2019; Li & Bastani, 2020; Wabersich & Zeilinger, 2018; Alshiekh et al., 2018); however, these approaches are not applicable to perception-based control.”
Provable repair of deep neural networks (2021, neutral): “Thus, multi-layer GaLU networks do not satisfy the key theoretical properties of DDNNs. Alshiekh et al. [3], Zhu et al. [61] ensure safety of re-inforcement learning controllers by synthesizing a shield guaranteeing it satisfies a temporal logic property.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “a finite-state automata) to guarantee (or bound the probability, in the stochastic case) that following a policy will not lead to an unsafe outcome [116, 106, 61, 57, 10].”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Many researchers have proposed extensions of the existing deep RL algorithms to address safety, such as learning an additional safety layer that projects raw actions to a safe feasibility set [3, 25, 20, 17], learning the boundary of the safe states with a classifier [38, 49, 23], expanding the iden”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, methodology): “This idea has been explored in [4, 11], which utilize Hamilton-Jacobi reachability analysis to deﬁne a task policy and safety controller, and in the context of shielding [2, 13, 21].”
Model-augmented Prioritized Experience Replay (2022, methodology): “For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed in parallel of methods for improving the sample efficiency of RL.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Based on formal methods [296], Safe MARL via shielding is proposed[80], in which they study safe MARL problem by one of formal methods, Linear Temporal Logic (LTL) [217] that is already adopted in safe single agent settings [8].”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “…the system in the safe set despite all possible model errors and process noise, captured by D . achieved through a safety ﬁlter (7) or shielding (117), ﬁnds the minimal modiﬁcation of a learning-based control input u learn (see Figure 2 ) such that the system’s state stays inside the set Ω safe…”
Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks (2020, methodology): “In all scenarios, we compare against the standard differential Q-learning approach that we refer to as the baseline and the same differential Q-learning approach where actions violating the given temporal logic formula are disallowed using the method introduced by Alshiekh et al. (2018).”
Neurosymbolic Programming (2021, background): “Extending prior work on shielded, or simplex, control (Alshiekh et al., 2018), the paper extends its safety guarantees to a certain class of neurosymbolic policies.”