Transparency and Explanation in Deep Reinforcement Learning Neural Networks

By Rahul Radhakrishnan Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, Katia P. Sycara. 2018

195 citations recorded by the source corpus.

Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to form coherent explanations of the system's decisions and actions. Transparency is important not only for user trust, but also for software debugging and certification. In recent years, Deep Neural Networks have made great advances in multiple application areas. However, deep neural networks are opaque. In this paper, we report on work in transparency in Deep Reinforcement Learning Networks (DRLN). Such networks have been extremely successful in accurately learning action control in image input domains, such as Atari games. In this paper, we propose a novel and general method that (a) incorporates explicit object recognition processing into deep reinforcement learning models, (b) forms the basis for the development of "object saliency maps", to provide visualization of internal states of DRLNs, thus enabling the formation of explanations and (c) can be incorporated in any existing deep reinforcement learning framework. We present computational results and human experiments to evaluate our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, methodology): “Functionally, this can be accomplished through a variety of means, including changing the color of the pixel (Simonyan et al., 2014), adding a gray mask (Zeiler & Fergus, 2014), blurring a small region (Greydanus et al., 2017), or masking objects with the background color (Iyer et al., 2018).”
Towards Better Interpretability in Deep Q-Networks (2018, methodology): “Perturbation based saliency maps using a continuous mask across the image and also using object segmentation based masks have been studied in the context of deep-RL (Greydanus et al. 2017; Iyer et al. 2018; Li, Sycara, and Iyer 2017).”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, methodology): “Shapley Additive Explanations (SHAP) tools [59], Saliency Map [60], and Gradient-weighted Class Activation Mapping (Grad-CAM) [61] are widely used model-agnostic explanation tools.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “For those methods, input feature-related interpretation algorithms, such as LIME and SmoothGrad, have already been explored for explaining DRL methods [15, 65, 80, 135].”
An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents (2018, background): “Another approach is to modify the DNN architecture or training procedure such that a trained model will have more interpretable features [18, 2].”