AI Safety Gridworlds

By Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, Shane Legg. 2017

298 citations recorded by the source corpus.

We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Alignment of Language Agents (2021, background): “AI safety gridworlds (Leike et al., 2017) also assumes a delegate agent, one which can physically move about in a gridworld, and doesn’t focus on safety in terms of language.”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “When phrased in terms of AI safety problems, this assumption states that we can learn to avoid various specification problems (Leike et al., 2017; Ortega et al., 2018) in practice.”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “• This system is robust to ’natural’ distributional shifts [49] [56];”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, background): “All these are concrete problems on AI safety, which has been recently studied in [146].”
Google Research Football: A Novel Reinforcement Learning Environment (2019, methodology): “search questions centered around a speciﬁc reinforcement learning set of topics. For instance, the DeepMind Control Suite (Tassa et al., 2018) focuses on continuous control, the AI Safety Gridworlds (Leike et al., 2017) on learning safely, whereas the Hanabi Learning Environment (Bard et al., 2019) ”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “What constitutes a ‘reasonable’ language is a long-standing open problem, see Leike et al. [2017] and M¨uller [2010].”
Good and safe uses of AI Oracles (2017, background): “ by the quantity of information it can transmit. 1 Introduction In recent years many have raised concerns about potential harms which might be caused by artiﬁcial intelligence (Amodei et al., 2016), (Leike et al., 2017). The risk is that the aim of these agents would not be aligned with human intere”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, unknown): “…we provide evidence that bounds on uncertainty and risk and may provide a useful tool for detecting reward hacking/gaming (Amodei et al., 2016), a common problem in reward inference from demonstrations (Ibarz et al., 2018) as well as reinforcement learning (Ng et al., 1999; Leike et al., 2017).”
Hidden Incentives for Auto-Induced Distributional Shift (2020, background): “This is an example of a speciﬁcation problem (Leike et al., 2017; Ortega et al., 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and “what we really meant”: to maximize accuracy via (1), which is difﬁcult to encode as a performance metric.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “1) AI Safety Gridworlds: AI Safety Gridworlds [72] 8 is a kind of 2-”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “A trustworthy visual recognition system should not only produce accurate predictions on known context, but also detect unknown examples and reject them (or hand them over to human users for safe handling) [1], [2], [3], [4], [5], [6], [7], [8].”
Reward Constrained Policy Optimization (2018, background): “As our experiments show, the coeﬃcient is not shared across domains, a coeﬃcient which leads to a satisfying behavior on one domain may lead to catastrophic failure on the other (issues also seen in Leike et al. (2017) and Mania et al. (2018)).”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, background): “Safety is crucial for reinforcement learning in many realworld settings [32], [33], [34].”
A Lyapunov-based Approach to Safe Reinforcement Learning (2018, methodology): “Motivated by the safety issues of RL in [22], we validate our safe RL algorithms using a stochastic 2D grid-world motion planning problem.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “We hope to help shrink the gap between the control and RL communities by creating a common vocabulary and introducing benchmarks for algorithm evaluation that can be leveraged by both (18, 19).”
Measuring and avoiding side effects using relative reachability (2018, methodology): “1 Irreversible Side Effects (Box) We test our set of penalties on the Irreversible Side Effects environment from the AI Safety Gridworlds suite (Leike et al., 2017), shown in Figure 4.”
Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration (2018, background): “Within the RL community this can be seen as a particular type of apprenticeship learning (Abbeel and Ng 2004) where the agent is learning how to be safe , rather than only maximizing reward (Leike et al. 2017).”
A Perspective on Objects and Systematic Generalization in Model-Based RL (2019, background): “In contrast, agents trained with Reinforcement Learning (RL) frequently fall short in this regard: they rely on excessive amounts of data (Franc¸ois-Lavet et al., 2018) and are unable to generalize beyond their initial training regime (Leike et al., 2017).”
Avoiding Side Effects By Considering Future Tasks (2020, background): “This environment [13] contains a box that needs to be pushed out of the way for the agent to reach the goal.”