Visual interpretability for deep learning: a survey

By Quanshi Zhang, Song-Chun Zhu. 2018

917 citations recorded by the source corpus.

This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, interpretability is always Achilles’ heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of a low interpretability of their black-box representations. We believe that high model interpretability may help people break several bottlenecks of deep learning, e.g., learning from a few annotations, learning via human–computer communications at the semantic level, and semantically debugging network representations. We focus on convolutional neural networks (CNNs), and revisit the visualization of CNN representations, methods of diagnosing representations of pre-trained CNNs, approaches for disentangling pre-trained CNN representations, learning of CNNs with disentangled representations, and middle-to-end learning based on model interpretability. Finally, we discuss prospective trends in explainable artificial intelligence.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “Most work on interpretability methods focuses on models for computer vision applications (e.g. Xu et al. (2015); Zhang & Zhu (2018)), whereas the interpretability of NLP models is apparently less studied.”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “7 7 Generally, many of the visualization methods are adapted from the vision domain, where they have been extremely popular; see Zhang and Zhu (2018) for a survey.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “A review of explanatory methods focused on understanding unit representations used by visual CNNs can be found in [52], which examines methods for visualization of CNN representations in intermediate network layers, diagnosis of these representations, disentanglement representation units, the…”
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, neutral): “[50] and DU et al.”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “Network interpretability: Recent studies of the interpretability of visual network are reviewed in [46].”
Interpreting CNNs via Decision Trees (2018, background): “[43] made a survey for the interpretability of deep visual models.”
Interpretable Convolutional Neural Networks (2017, methodology): “Our previous paper [39] provides a comprehensive survey of recent studies in exploring visual interpretability of neural networks, including 1) the visualization and diagnosis of CNN representations, 2) approaches for disentangling CNN representations into graphs or trees, 3) the learning of CNNs wi”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “85 This research still lacks a deeper appreciation of how humans, with various levels of background knowledge and differing roles and goals, interpret different explanations and uncertainty information.”
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation (2018, methodology): “A review of visualization methods for interpreting deep convolutional neural nets is provided in [9].”
Artificial intelligence in Cancer Imaging: Clinical Challenges and Applications (2019, background): “209 The benefits of trust and transparency in AI systems will differ based on their performance, allowing for the identification of failures when AI is subhuman and, consequently, transforming superhuman AI into a learning resource.”
A Shared Attention Mechanism for Interpretation of Neural Automatic Post-Editing Systems (2018, background): “In deep learning, highly interpretable models can help researchers to overcome outstanding issues such as learning from fewer annotations, learning with human-computer interactions and debugging network representations (Zhang and Zhu, 2018).”
Machine behaviour (2019, background): “They are fed input and produce output, but the exact functional process for generating this output is hard to interpret even to the very scientists who generate the algorithms themselves 44 , though some progress in interpretability is being made 45,46 .”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Explanators generating salient masks were investigated in [20, 30] whilst [20, 39] reviewed methods that graphically represent the inner structure and functioning of neural networks with flow-charts or other explanatory graphs.”
Extracting Possibly Representative COVID-19 Biomarkers from X-ray Images with Deep Learning Approach and Image Data Related to Pulmonary Diseases (2020, background): “The second issue is related to the flaws of Deep Learning and is often referred to as the issue of interpretability [6].”
Explainable Agents and Robots: Results from a Systematic Literature Review (2019, background): “Nevertheless, recent studies on the explanation methods mainly focused on the data-driven algorithms aiming to interpret the results of “black-box” machine learning mechanisms such as deep neural networks (DNN) [108].”