Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
By Akanksha Atrey, Kaleigh Clary, David D. Jensen. 2019
113 citations recorded by the source corpus.
Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.
Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dhruv Batra
- Mohit Bansal
- Trevor Darrell
- Abhinav Kumar Gupta
- Jitendra Malik
- Vladlen Koltun
- Sergey Levine
- Kristen Grauman
In Reinforcement Learning
- Dhruv Batra
- Devendra Singh Chaplot
- Jitendra Malik
- Manolis Savva
- Vladlen Koltun
- Sergey Levine
- F. Meier
- Aaron Gokaslan
In Safety, Robustness & Interpretability
- Lesia Semenova
- Zhi Chen
- Chudi Zhong
- Haiyang Huang
- Cynthia Rudin
- Jitendra Malik
- Vladlen Koltun
- Zsolt Kira
Citations in the corpus
- Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[103] Akanksha Atrey, Kaleigh Clary, and David Jensen.”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “[18] have argued that for deep reinforcement learning, saliency maps should be used for exploratory, and not explanatory, purposes.”
- Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “For those methods, input feature-related interpretation algorithms, such as LIME and SmoothGrad, have already been explored for explaining DRL methods [15, 65, 80, 135].”