Visualizing and Understanding Atari Agents

By Sam Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern. 2017

406 citations recorded by the source corpus.

While deep reinforcement learning (deep RL) agents are effective at maximizing rewards, it is often unclear what strategies they use to do so. In this paper, we take a step toward explaining deep RL agents through a case study using Atari 2600 environments. In particular, we focus on using saliency maps to understand how an agent learns and executes a policy. We introduce a method for generating useful saliency maps and use it to show 1) what strong agents attend to, 2) whether agents are making decisions for the right or wrong reasons, and 3) how agents evolve during learning. We also test our method on non-expert human subjects and find that it improves their ability to reason about these agents. Overall, our results show that saliency information can provide significant insight into an RL agent's decisions and learning behavior.

Subfields: Artificial Intelligence, Safety Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…state space of agents using t-SNE plots (Zahavy et al., 2016; Jaderberg et al., 2018), examining what agents attend to when they make decisions (Greydanus et al., 2018), evaluating models’ sensitivity to the presence/intensity of high-level human concepts (Kim et al., 2017), optimizing a model…”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “We choose to use Gaussian blurring to perturb the irrelevant regions, which is also used in a previous Explainable RL work [10].”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, methodology): “The Reason column represents whether an explanation for agent behavior was provided (Y/N) and the Exp column represents whether an experiment was conducted to evaluate the explanation (Y/N).”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, methodology): “Previous efforts to visualize deep RL agents [1, 2, 3] focus on generating saliency maps to understand the magnitude of policy changes as a function of a perturbation of the input.”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “Since SpRAy aims to investigate classi ﬁ er behavior, it is applied to the heatmaps and not to the raw images (see Supplementary Figures 20, 24”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “We use the method proposed by Greydanus et al. (2018), which takes a stack of 4 frames and passes a 3x3 mask over each of the frames with a stride of 1.”
Towards Better Interpretability in Deep Q-Networks (2018, methodology): “Commonly used visualization methods for deep learning such as saliency maps and t-SNE plots of embeddings have been applied to deep RL models (Greydanus et al. 2017; Zahavy, Ben-Zrihem, and Mannor 2016; Mnih et al. 2015).”
Explainability in Deep Reinforcement Learning (2020, background): “Transparent algorithms are well known and used in standard Machine Learning (e.g., linear regression, decision trees or rule-based systems).”
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems (2021, unknown): “While STSTs – be they saliency regions over images, or motion trajectories – have been used in the machine learning community as a means to either advice or interpret the operation of AI systems (Greydanus et al. 2018; Zhang et al. 2020; Christiano et al. 2017), we contend that they will not scale t”
An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents (2018, methodology): “While much more visualization and understanding work has been done for vision models than for DRL, a few papers directly focus on understanding DRL agents [Greydanus et al., 2017; Zahavy et al., 2016], and many others feature some analysis of DRL agent behavior (often in the form of t-SNE diagrams o”
Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, background): “For example, ALE does not enable testing the conjecture that agents trained on Breakout learn to build tunnels (Mnih et al., 2015) or that they enter a tunneling mode (Greydanus et al., 2018).”
Free-Lunch Saliency via Attention in Atari Agents (2019, methodology): “playing Atari games; later, Jacobians of value and advantage streams with respect to input images were used for the same purpose in [40]. The work [44] investigated t-SNE embeddings in more depth. In [15], saliency maps for Atari agents trained with A3C were visualized by blurring different parts of”
Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents (2019, methodology): “[29] also use t-SNE embeddings for visualization, while [11] examine how the current state affects the policy in a vision-based approach using saliency methods .”
Learn to Interpret Atari Agents (2018, methodology): “Interested readers can refer to [19], [20] for more details.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “For those methods, input features related interpretation algorithms, such as LIME and SmoothGrad, have already been explored for explaining DRL methods [15, 65, 80, 135].”
Learning Finite State Representations of Recurrent Policy Networks (2018, result): “There have been other recent efforts towards understanding Atari agents (Zahavy et al., 2016; Greydanus et al., 2017).”
Observational Overfitting in Reinforcement Learning (2019, methodology): “As shown in Figure 1, by using saliency maps (Greydanus et al., 2018), we found that the agent strongly overﬁts to the scoreboard/timer (i.e. an artifact correlated with progress in the level through determinism).”
Counterfactual States for Atari Agents via Generative Deep Learning (2019, background): “The majority of these techniques provide explanations as to why the agent chose a particular action (e.g. [Greydanus et al. , 2018]).”
Sidekick Policy Learning for Active Visual Exploration (2018, background): “that are important for selecting an action [18].”