Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference

By Reza Ghaeini, Xiaoli Z. Fern, Prasad Tadepalli. 2018

99 citations recorded by the source corpus.

Deep learning models have achieved remarkable success in natural language inference (NLI) tasks. While these models are widely explored, they are hard to interpret and it is often unclear how and why they actually work. In this paper, we take a step toward explaining such deep learning based models through a case study on a popular neural model for NLI. In particular, we propose to interpret the intermediate layers of NLI models by visualizing the saliency of attention and LSTM gating signals. We present several examples for which our methods are able to reveal interesting insights and identify the critical information contributing to the model decisions.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Saliency can also be computed with respect to intermediate values, rather than input features (Ghaeini et al., 2018).”
Attention is not Explanation (2019, background): “Comparing such importance measures to attention scores may provide additional insights into the working of attention based models (Ghaeini et al., 2018).”
Is Attention Interpretable? (2019, methodology): “Since attention calculates a distribution over inputs, prior work has used attention as a tool for interpretation of model decisions (Wang et al., 2016; Lee et al., 2017; Lin et al., 2017; Ghaeini et al., 2018).”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, methodology): “Many works [15, 35, 16] suggest that attention weight assignment in encoder-decoder attention plays a role similar to word alignment in traditional approaches [1, 7, 27, 5].”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “mechanisms have been considered as “inherently interpretable” (Ghaeini et al., 2018; Lee et al., 2017), recent work cast doubt regarding their faithfulness (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
Understanding Multi-Head Attention in Abstractive Summarization (2019, background): “, 2015) has gained popularity as a means of obtaining insight in the inner workings of deep neural networks (e.g., Lei, 2017; Choi et al., 2016; Gilpin et al., 2018; Ghaeini et al., 2018).”
A Survey of the State of Explainable AI for Natural Language Processing (2020, background): “It is possible to utilize (and even combine) other operations presented here to interpret gating signals to aid feature importance explainability (Ghaeini et al., 2018).”
Interrogating the Explanatory Power of Attention in Neural Machine Translation (2019, methodology): “Saliency methods use the gradient of the inputs to define importance (Li et al., 2016a; Ghaeini et al., 2018; Ding et al., 2019).”
Attention Interpretability Across NLP Tasks (2019, background): “Apart from substantial performance beneﬁt (Vaswani et al., 2017), attention also provides interpretability to neural models (Wang et al., 2016; Lin et al., 2017; Ghaeini et al., 2018) which are usually criticized for being black-box function approximators (Chakraborty et al., 2017).”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “ghlights, where the interface highlights spans of text that are most salient to the model’s prediction directly in the input text, so the user need not go back and forth between input and explanation [18, 36, 42, 62]. Figure2shows one example beer review used in our study. We also experimented with ”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, background): “Koehn and Knowles (2017) showed that attention-based word alignment interpretation may be subject to “off-by-one” errors.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Attention and gating are frequently used in NLP models, and there have been plenty of work aiming to understand the model through these weights [2, 40, 111, 112] and investigate the reliability of using them as explanations [55, 103, 129].”
Emergent Structures and Training Dynamics in Large Language Models (2022, methodology): “To address these limi-653 tations, Ghaeini et al. (2018) examine the saliency of attention and LSTM gating signal in the inter-655 mediate layers of ESIM models, an architecture designed for NLI tasks (Chen et al., 2017).”
Human-grounded Evaluations of Explanation Methods for Text Classification (2019, methodology): “Besides, there are other explanation methods designed for specific deep learning architectures such as attention mechanism (Ghaeini et al., 2018) and extractive rationale generation (Lei et al.”