Automatic Rule Extraction from Long Short Term Memory Networks

By W. James Murdoch, Arthur Szlam. 2016

95 citations recorded by the source corpus.

Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, neutral): “…(Springenberg et al., 2014; Sundararajan et al., 2017; Selvaraju et al., 2016; Baehrens et al., 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al., 2016;…”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “[40] succeeded in extracting rules from an LSTM by applying a decompositional approach.”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, background): “, 2016), or to general deep neural networks for text (Denil et al., 2014; Li et al., 2016a; Arras et al., 2016a; Li et al., 2016b; Murdoch and Szlam, 2017).”
Interpretable and Steerable Sequence Learning via Prototypes (2019, methodology): “Recently, model distillation is also used to explain deep sequence models [5, 23, 26, 27].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “Murdoch and Szlam [358] showed that the output of an LSTM can be decomposed into a product of factors, where each factor can be interpreted as the contribution at a particular timestep.”
Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs (2018, methodology): “Murdoch & Szlam (2017) introduced a decomposition of the LSTM’s output embedding into a sum over word coefﬁcients, and demonstrated that those coefﬁcients are meaningful by using them to distill LSTMs into rules-based classiﬁers.”
Visualizing and Understanding Atari Agents (2017, background): “While an abundance of literature has addressed techniques for explaining deep image classifiers [6, 19, 23, 28] and deep sequential models [10, 18], very little work has been done to explain deep RL agents.”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “Besides back-propagation and attention-based methods discussed in the introduction, another class of interpretation methods (Murdoch and Szlam, 2017; Singh et al., 2018; Jin et al., 2020) builds on prior work in cooperative game theory (e.g., Shapley value of Shapley (1953)).”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Word Importance Scores [253] visualizes the importance of specific inputs for determining the output of a LSTM.”
Learning Finite State Representations of Recurrent Policy Networks (2018, background): “There have been efforts made in the past to understand the internals of Recurrent Networks (Karpa-thy et al., 2015; Arras et al., 2017; Strobelt et al., 2016; Murdoch & Szlam, 2017; Jacobsson, 2005; Omlin & Giles, 1996).”
Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “To extract rational evidence from the entire inputs, one either leverages a particular explanation method (Lei et al., 2016; Wang et al., 2019) or an auxiliary evidence extraction model (Murdoch and Szlam, 2017; Arras et al., 2017).”
Interpretable Deep Learning under Fire (2018, background): “Existing work has exploited interpretability to debug DNNs [50], dig st security analysis results [25], and detect adversarial inputs [38, 72].”
Input Switched Affine Networks: An RNN Architecture Designed for Interpretability (2016, methodology): “Con-currently with our submission, Murdoch & Szlam (2017) decomposed LSTM outputs using telescoping sums of statistics computed from memory cells at different RNN steps.”
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples (2017, background): “Indeed, several lines of work attempt to glimpse into the RNN black-box [25,24,32,33,20,23,22,30,4,7,21,19,28,1,18].”
Exploring Interpretable LSTM Neural Networks over Multi-Variable Data (2019, methodology): “As for post-analyzing interpretation, Murdoch et al. (2018); Murdoch & Szlam (2017); Arras et al. (2017) extracted temporal importance scores over words or phrases of individual sequences by decomposing the memory cells of trained RNNs.”
Understanding Hidden Memories of Recurrent Neural Networks (2017, methodology): “We ﬁrst show that the numerator of probability p i in Equation 2 can be decomposed into a product of factors using the method proposed by Murdoch and Szlam [31] (we always set hhh ( 0 ) = 0 0 0): (3) Here, exp ( uuu Ti ∆ hhh ( t ) ) can be interpreted as the multiplicative contribution of word t to ”