Graying the black box: Understanding DQNs

By Tom Zahavy, Nir Ben-Zrihem, Shie Mannor. 2016

292 citations recorded by the source corpus.

In recent years there is a growing interest in using deep representations for reinforcement learning. In this paper, we present a methodology and tools to analyze Deep Q-networks (DQNs) in a non-blind matter. Using our tools we reveal that the features learned by DQNs aggregate the state space in a hierarchical fashion, explaining its success. Moreover we are able to understand and describe the policies learned by DQNs for three different Atari2600 games and suggest ways to interpret, debug and optimize deep neural networks in reinforcement learning.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “This includes preliminary work on visualizing the latent state space of agents using t-SNE plots (Zahavy et al., 2016; Jaderberg et al., 2018), examining what agents attend to when they make decisions (Greydanus et al., 2018), evaluating models’ sensitivity to the presence/intensity of high-level…”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “. See Section 4 for a complete description. Network features - Since the policy or value networks of an agent learn to extract taskrelevant high-level features of the environment (Mnih et al., 2015; Zahavy et al., 2016; Silver et al., 2016) they can be useful quantities for the agent to learn to con”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “’s deep Q-learning model [210], Zahavy [329] showed that the network automatically discovers hierarchical structures.”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “…the operation of deep learning networks, with an audience of other researchers (e.g., Goyal et al., 2016abc; Karpathy et al., 2015; Kim, et al., 2016; Moeyersoms, et. al., 2016; Sadeghi, et al., 2015; Selvaraju, et al., 2017; Yosinski, et al., 2015; Zahavy et al., 2016; Zeiler and Fergus, 2014).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “Breakout Perturbation N N Zahavy et al. (2016) “A diver is noticed in the saliency map but misun-derstood as an enemy and being shot at.””
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “Explanation methods were successfully applied to a wide set of complex real-world problems such as the analysis of faces [81, 123, 124], EEG and fMRI data [23, 50], human gait [48], videos [125], speech [126], reinforcement learning [25], biological data [127], text [26,128,129], or comparing human ”
Programmatically Interpretable Reinforcement Learning (2018, background): “Many recent efforts in deep learning aim to make deep networks more interpretable (Montavon et al., 2017; Lipton, 2016; Garnelo et al., 2016; Zahavy et al., 2016; Shanahan, 2005; Lake et al., 2016).”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “Recently, it has been shown that Deep Networks tend to implicitly capture the hierarchical composition of a given task (Zahavy et al., 2016).”
Active Long Term Memory Networks (2016, neutral): “This procedure allows the creation of representations that DQNs transverse quasi-hierarchically [33] during play.”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, methodology): “Mnih et al. (2015) performed a simple clustering analysis to understand the state dynamics within DQN. Zahavy et al. (2016) also used clustering methods within a simple graphical interface to study emerging state abstractions in DQN networks, and further visualized the saliency of various stimuli.”
Visualizing and Understanding Atari Agents (2017, methodology): “Past methods include t-SNE embeddings [15, 26], Jacobian saliency maps [25, 26], and reward curves [15].”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “agents [211, 212]).”
Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention (2017, methodology): “[30] proposes a visualization method to interpret the agents action by describing Markov Decision Process model as a directed graph on a t-SNE map.”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “This success is mainly attributed to the power of deep neural networks to learn rich domain representations for approximating the value function or policy (Mnih et al., 2015; Zahavy et al., 2016; Zrihem et al., 2016).”
Learning Finite State Representations of Recurrent Policy Networks (2018, result): “There have been other recent efforts towards understanding Atari agents (Zahavy et al., 2016; Greydanus et al., 2017).”