Reasoning about Entailment with Neural Attention

By Tim Rocktaschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Phil Blunsom. 2015

779 citations recorded by the source corpus.

While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases. Furthermore, we present a qualitative analysis of attention weights produced by this model, demonstrating such reasoning capabilities. On a large entailment dataset this model outperforms the previous best neural model and a classifier with engineered features by a substantial margin. It is the first generic end-to-end differentiable system that achieves state-of-the-art accuracy on a textual entailment dataset.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, neutral): “2 LSTM with word-by-word attention (Rockt¨aschel et al., 2016) 83 .”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “encode the premise and the hypothesis respectively, and apply neural attention to reason about their logical relationship (Rocktäschel et al., 2016; Wang and Jiang, 2016).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “, 2017), and natural language inference (Rocktäschel et al., 2016).”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “The dominating trend in these models is to build complex, deep text representation models, for example, with convolutional networks (LeCun et al., 1990, CNNs henceforth) or long short-term memory networks (Hochreiter and Schmidhuber, 1997, LSTMs henceforth) with the goal of deeper sentence comprehen”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “The recent spurt of powerful end-to-end-trained neural networks for Natural Language Processing (Hermann et al., 2015; Rockt¨aschel et al., 2016; Weston et al., 2015, a.o.) has sparked interest in tasks to measure the progress they are bringing about in genuine language understanding.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “While previous work shows that performance on the NLI task can be improved by using cross-sentence attention between the premise and hypothesis (Rocktäschel et al., 2016; Gong et al., 2018), we focus on methods with fixed-size sentence embeddings.”
Quantifying Attention Flow in Transformers (2020, background): “…the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark et al., 2019).”
e-SNLI: Natural Language Inference with Natural Language Explanations (2018, background): “Attention-based models, such as [2, 25], offer some degree of interpretability and have been shown to also improve performance on downstream tasks.”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “For instance, Sanchez et al. (2018) and Glockner et al. (2018) extracted examples from SNLI (Bow-man et al., 2015) and replaced speciﬁc words such as hypernyms, synonyms, and antonyms, followed by manual veriﬁcation.”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, methodology): “Seo et al. (2016) introduce bi-directional attention flow networks to model question-passage pairs at multiple levels of granularity.”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, neutral): “…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
Enhanced LSTM for Natural Language Inference (2016, methodology): “(8) 100D LSTM with attention (Rocktäschel et al., 2015) 250K 85.”
Rationalizing Neural Predictions (2016, methodology): “Such models have been successfully applied to many NLP problems, improving both prediction accuracy as well as visualization and interpretability (Rush et al., 2015; Rocktäschel et al., 2016; Hermann et al., 2015).”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “A more fine-grained attention mechanism is proposed in [16].”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “Since then, several neural-network based methods have been proposed over SNLI that either use sentence encoding models to individually encode hypothesis and premise or attention based models that encode the sentences together and align similar words in hypothesis and premise [8, 50, 62, 59].”
Order-Embeddings of Images and Language (2015, methodology): “The state-of-the-art method for 3-class classiﬁcation on SNLI is that of Rockt¨aschel et al. (2015).”
Learning Natural Language Inference with LSTM (2015, methodology): “While there are different variations of LSTMs, here we present the one adopted by Rocktäschel et al. (2016). Specifically, let us use X = (x1,x2, .”
Natural Language Inference over Interaction Space (2017, methodology): “., 2016; Vendrov et al., 2015; Mou et al., 2015; Liu et al., 2016; Munkhdalai & Yu, 2016). (ii) Joint feature models which use the cross sentence feature or attention from one sentence to another(Rocktaschel et al., 2015; Wang & Jiang, 2015; Cheng et al., 2016; Parikh et al., 2016; Wang¨ et ”