Improving Interpretability of Deep Neural Networks with Semantic Information

By Yinpeng Dong, Hang Su, Jun Zhu. 2017

133 citations recorded by the source corpus.

Interpretability of deep neural networks (DNNs) is essential since it enables users to understand the overall strengths and weaknesses of the models, conveys an understanding of how the models will behave in the future, and how to diagnose and correct potential problems. However, it is challenging to reason about what a DNN actually does due to its opaque or black-box nature. To address this issue, we propose a novel technique to improve the interpretability of DNNs by leveraging the rich semantic information embedded in human descriptions. By concentrating on the video captioning task, we first extract a set of semantically meaningful topics from the human descriptions that cover a wide range of visual concepts, and integrate them into the model with an interpretive loss. We then propose a prediction difference maximization algorithm to interpret the learned features of each neuron. Experimental results demonstrate its effectiveness in video captioning using the interpretable features, which can also be transferred to video action recognition. By clearly understanding the learned features, users can easily revise false predictions via a human-in-the-loop procedure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In the task of video captioning, the authors in [111] use a CNN model combined with a bi-directional LSTM model as encoder to extract video features and then feed these features to an LSTM decoder to generate textual descriptions.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “as part of the loss function [77], or as part of the architecture structure, in terms of operations between layers [78], [79].”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Dong et al. (2017b) provide an interpretive loss function to increase the visual ﬁdelity of the learned features.”
Deep Reinforcement Learning: An Overview (2017, background): “ciousness prior,Bengio(2017) ML with knowledge,Song and Roth(2017) causality,Pearl(2018),Johansson et al.(2016) interpretability,Zhang and Zhu(2018) surveyed visual interpretability for deep learning,Dong et al. (2017) George et al.(2017) Yang and Mitchell(2017) 4 IMPORTANT MECHANISMS In this sectio”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Various approaches [46, 114, 140, 191] have been proposed to improve the”
Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges (2022, unknown): “Gaining an insight into how and why the SE model arrives at a particular decision or outcome not only builds confidence in the model to deal with unknown situations, thereby reducing the risk of uncertainty, but also helps to understand the overall strengths and weaknesses of the models and guides i”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “As a result, there have been efforts in defining and evaluating interpretable approaches in the superviselearning domain [179]–[181].”
Regional Multi-Scale Approach for Visually Pleasing Explanations of Deep Neural Networks (2018, methodology): “improved the interpretability of a videocaptioning CNN-RNN model by embedding the prediction difference maximization operation in the model [15].”
Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples (2017, methodology): “Unlike other methods that use high-level semantic attributes to improve the interpretability of DNNs explicitly [4], we train the DNNs towards interpretability implicitly.”
Decoding the Deep: Exploring Class Hierarchies of Deep Representations Using Multiresolution Matrix Factorization (2017, methodology): “The networks are not trained to learn the hierarchy of categories (unlike the alternate works from [33, 7, 2]) – the”