Explaining Predictions of Non-Linear Classifiers in NLP

By Leila Arras, Franziska Horn, Grgoire Montavon, K. Müller, Wojciech Samek. 2016

123 citations recorded by the source corpus.

Layer-wise relevance propagation (LRP) is a recently proposed technique for explaining predictions of complex non-linear classifiers in terms of input variables. In this paper, we apply LRP for the first time to natural language processing (NLP). More precisely, we use it to explain the predictions of a convolutional neural network (CNN) trained on a topic categorization task. Our analysis highlights which words are relevant for a specific prediction of the CNN. We compare our technique to standard sensitivity analysis, both qualitatively and quantitatively, using a "word deleting" perturbation experiment, a PCA analysis, and various visualizations. All experiments validate the suitability of LRP for explaining the CNN predictions, which is also in line with results reported in recent image classification studies.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, background): “For instance, the authors of [20, 2, 3] observed that models may have the same classiﬁcation performance, but largely differ in terms of what features they use as the basis for their decisions.”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “, 2016), or to general deep neural networks for text (Denil et al., 2014; Li et al., 2016a; Arras et al., 2016a; Li et al., 2016b; Murdoch and Szlam, 2017).”
Explaining nonlinear classification decisions with deep Taylor decomposition (2015, methodology): “2See [42, 43] for the application of decomposition techniques to text and EEG data.”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “Explanation methods were successfully applied to a wide set of complex real-world problems such as the analysis of faces [81, 123, 124], EEG and fMRI data [23, 50], human gait [48], videos [125], speech [126], reinforcement learning [25], biological data [127], text [26,128,129], or comparing human ”
Is Attention Interpretable? (2019, background): “For example, recent work has focused on which training instances (Koh and Liang, 2017) or which human-interpretable features were most relevant for a particular decision (Ribeiro et al., 2016; Arras et al., 2016).”
"What is relevant in a text document?": An interpretable machine learning approach (2016, methodology): “We furthermore extend previous work from [28] by adding a BoW/SVM baseline to the experiments and proposing a new criterion for assessing model explanatory power.”
Interpreting Neural Networks with Nearest Neighbors (2018, methodology): “We combine D K NN with CNN and LSTM models on six NLP text classiﬁcation tasks, including sentiment analysis and textual entailment, with no loss in classiﬁcation accuracy (Section 4).”
Towards Understanding Neural Machine Translation with Word Importance (2019, methodology): “There exists another straightforward method, Era-sure (Alvarez-Melis and Jaakkola, 2017; Arras et al., 2016; Zintgraf et al., 2017), which directly evaluates the word importance by measuring the translation performance degradation of each word.”
Interpreting the Predictions of Complex ML Models by Layer-wise Relevance Propagation (2016, methodology): “In [16] it was used for the identification of relevant words in text documents and in [17] for visualizing facial features related to age, happiness and attractivity.”
Identifying Individual Facial Expressions by Deconstructing a Neural Network (2016, background): “For a more theoretical view on LRP we refer the reader to [18] and for applications beyond image classiﬁcation to [1,23].”
DeepClue: Visual Interpretation of Text-Based Deep Stock Prediction (2019, methodology): “Many existing methods to extract such scores have been introduced in both the computer vision domain [9], [15], [16], [38], [39] and the natural language processing research community [40].”
Interpretable human action recognition in compressed domain (2017, methodology): “While these methods are limited in their range of applicability, general explanation techniques such as Layer-wise Relevance Propagation (LRP) [16] and Deep Taylor [17] have been recently introduced and applied to image, text and biomedical signal classification problems [18, 19, 20].”
Explainable Sentiment Analysis with Applications in Medicine (2018, methodology): “For example, LRP model [8] was also applied in order to extract from text which words were most relevant for a CNN’s classifier trained on a topic categorization task [65].”
VisualBackProp: efficient visualization of CNNs (2018, methodology): “The LRP technique was extended to Fisher Vector classifiers [13] and also used to explain predictions of CNNs in NLP applications [14].”
VisualBackProp: visualizing CNNs for autonomous driving (2016, methodology): “The LRP technique was later extended to Fisher Vector classifiers [16] and also used to explain predictions of CNNs in NLP applications [17].”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Layer-wise Relevance Propagation (LRP) and Sensitivity Analysis were tested in [327, 328, 329, 330] by removing important words from input documents in text classification problems [327, 328] or replacing the most relevant pixels (in case of images) by randomly sampling new pixel values from a unifo”