"What is relevant in a text document?": An interpretable machine learning approach

By Leila Arras, Franziska Horn, Grgoire Montavon, K. Müller, Wojciech Samek. 2016

300 citations recorded by the source corpus.

Text documents can be described by a number of abstract concepts such as semantic category, writing style, or sentiment. Machine learning (ML) models have been trained to automatically map documents to these abstract concepts, allowing to annotate very large text collections, more than could be processed by a human in a lifetime. Besides predicting the text’s category very accurately, it is also highly desirable to understand how and why the categorization process takes place. In this paper, we demonstrate that such understanding can be achieved by tracing the classification decision back to individual words using layer-wise relevance propagation (LRP), a recently developed technique for explaining predictions of complex non-linear classifiers. We train two word-based ML models, a convolutional neural network (CNN) and a bag-of-words SVM classifier, on a topic categorization task and adapt the LRP method to decompose the predictions of these models onto words. Resulting scores indicate how much individual words contribute to the overall classification decision. This enables one to distill relevant information from text documents without an explicit semantic information extraction step. We further use the word-wise relevance scores for generating novel vector-based document representations which capture semantic information. Based on these document vectors, we introduce a measure of model explanatory power and show that, although the SVM and CNN models perform similarly in terms of classification accuracy, the latter exhibits a higher level of explainability which makes it more comprehensible for humans and potentially more useful for other applications.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Information Extraction

Citations in the corpus

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Another approach identifies important input features that contribute to a model’s prediction via saliency methods (Li et al., 2016; Arras et al., 2017; Murdoch et al., 2018), which also typically ignore the model’s internal structure, although they may in principle be computed with respect to intern”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “The important or salient features can then be visualized in selected examples (Li et al., 2016a; Aubakirova and Bansal, 2016; Sundararajan et al., 2017; Arras et al., 2017a,b; Ding et al., 2017; Mur-doch et al., 2018; Mudrakarta et al., 2018; Mon-tavon et al., 2018; Godin et al., 2018).”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, neutral): “[12] [35] [36] [37] [38] [39] [40] use Layer-wise Relevance Propagation (LRP) to construct heat-maps/relevance-maps for interpretability.”
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, background): “For instance, the authors of [20, 2, 3] observed that models may have the same classiﬁcation performance, but largely differ in terms of what features they use as the basis for their decisions.”
Methods for interpreting and understanding deep neural networks (2017, methodology): “Arras et al. [3] considered a document classiﬁcation task on the 20-Newsgroup dataset, and compared the explanations of a convolutional neural network (CNN) trained on word2vec inputs to the explanations of a support vector machine (SVM) trained on bag-of-words (BoW) document representations.”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “…models such as deep convolutional neural networks for vision (Simonyan et al., 2014; Zeiler and Fergus, 2014; Bach et al., 2015; Ribeiro et al., 2016), or to general deep neural networks for text (Denil et al., 2014; Li et al., 2016a; Arras et al., 2016a; Li et al., 2016b; Murdoch and Szlam, 2017).”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “For example, for a probabilistic classification model, we can obscure or remove features, measure the drop in probability of the predicted class, and compare against the interpreter’s own prediction of relevance [17, 3].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “Most automatic evaluations of faithfulness measure the impact of perturbing or erasing words or tokens identified as important on model output (Arras et al., 2017; Montavon et al., 2017; Serrano and Smith, 2019; Samek et al., 2016; Jain and Wallace, 2019).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “One method proposed to do so is erasure , where the “most relevant” parts of the input—according to the explanation—are erased from the input, in expectation that the model’s decision will change (Arras et al., 2016; Feng et al., 2018; Serrano and Smith, 2019).”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, methodology): “At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis (2018, background): “More recent developments include DeepLIFT (Shrikumar et al., 2016) (not yet applicable to RNNs), layerwise relevance propagation (Bach et al., 2015) (only very recently adapted to textual input and LSTMs (Arras et al., 2017a,b)), and LIME (Ribeiro et al., 2016).”
Offensive Language Detection Explained (2020, methodology): “It is based on an idea by Arras et al. (2017).”
Human-grounded Evaluations of Explanation Methods for Text Classification (2019, methodology): “Arras et al. (2017a) and Xiong et al. (2018) used relevance scores generated by explanation methods to construct document vectors by weighted-averaging word vectors and checked how well traditional machine learning techniques manage these document vectors.”
Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities (2021, methodology): “Furthermore, when comparing models with similar performance, XAI can help in the selection by revealing the features that the models use to produce their decisions [14, 15].”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “Another approach identiﬁes important input features that contribute to a model’s prediction via saliency methods (Li et al., 2016; Arras et al., 2017; Murdoch et al., 2018), which also typically ignore the model’s internal structure, although one could compute them with respect to internal…”