Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention

By Yang Liu, Chengjie Sun, Lei Lin, Xiaolong Wang. 2016

285 citations recorded by the source corpus.

In this paper, we proposed a sentence encoding-based model for recognizing text entailment. In our approach, the encoding of sentence is a two-stage process. Firstly, average pooling was used over word-level bidirectional LSTM (biLSTM) to generate a first-stage sentence representation. Secondly, attention mechanism was employed to replace average pooling on the same sentence for better representations. Instead of using target sentence to attend words in source sentence, we utilized the sentence's first-stage representation to attend words appeared in itself, which is called "Inner-Attention" in our paper . Experiments conducted on Stanford Natural Language Inference (SNLI) Corpus has proved the effectiveness of "Inner-Attention" mechanism. With less number of parameters, our model outperformed the existing best sentence encoding-based approach by a large margin.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Liu et al. (2016a) proposes a sentence level attention which has a similar motivation but done differently.”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “The self-attentive sentence encoder (Liu et al., 2016; Lin et al., 2017) uses an attention mechanism over the hidden states of a BiLSTM to generate a representation u of an input sentence.”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “A softmax layer over the candidate answers is used to train the model with cross-entropy loss. d QAC OMPARE This model is similar to an NLI model from Liu et al. (2016).”
Enhanced LSTM for Natural Language Inference (2016, background): “A variety of more advanced networks have been developed since then (Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2016; Liu et al., 2016; Munkhdalai and Yu, 2016a; Rocktäschel et al., 2015; Wang and Jiang, 2016; Cheng et al., 2016; Parikh et al., 2016; Munkhdalai and Yu, 2016b; Sha et al., ”
Natural Language Inference over Interaction Space (2017, methodology): “BiLSTM intra-attention encoders(Liu et al., 2016) 84.”
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2019, background): “The SNLI corpus [3] of ∼ 570 K such human-written instances enabled a plethora of works on this task [28, 25, 22].”
Bilateral Multi-Perspective Matching for Natural Language Sentences (2017, neutral): “…cosine matching func-Models Accuracy [Bowman et al. , 2015] 77.6 [Vendrov et al. , 2015] 81.4 [Mou et al. , 2015] 82.1 [Rockt¨aschel et al. , 2015] 83.5 [Liu et al. , 2016b] 85.0 [Liu et al. , 2016a] 85.1 [Wang and Jiang, 2015] 86.1 [Cheng et al. , 2016] 86.3 [Parikh et al. , 2016] 86.8…”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, methodology): “Besides, attention based sentence encoding with mean pooling (Liu et al. 2016) is proposed to automatically focus a significant part of input sequence then generates more effective sentence representation.”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, background): “A possible solution could be to condition the attention on the inputs only, for instance so-called self-attention (Liu et al., 2016) or inner-attention (Lin et al., 2017).”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, methodology): “Similar to the attention from multiple perspectives in multi-head attention, multi-perspective context matching (Wang et al., 2016) explores the dependencies between passage and question from multiple perspectives for reading comprehension, while self-attentive structure (Lin et al., 2017) embeds…”
Are BLEU and Meaning Representation in Opposition? (2018, methodology): “We adopt an alternative approach, which is to use inner attention 1 (Liu et al., 2016; Li et al., 2016) to compute several weighted averages of the encoder states (Lin et al., 2017).”
Neural Machine Translation: A Review (2019, background): “tasks such as sentiment analysis [116], natural language inference [62, 96, 117, 118], text summarization [119], headline generation [120], sentence embedding [63, 64, 121], and reading comprehension [122].”
On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (2018, background): “5 We follow the common practice of feeding the concatenation ( v , u ) ∈ R 4 d to a classiﬁer (Rocktäschel et al., 2016; Bowman et al., 2015; Mou et al., 2016; Liu et al., 2016; Cheng et al., 2016; Munkhdalai and Yu, 2017).”
Lifelong Learning of Hate Speech Classification on Social Media (2021, methodology): “We compare our methods with the following methods: Fine-tuning: The model contains two bidirectional LSTM encoders (Hochreiter and Schmid-huber, 1997; Zhou et al., 2016; Liu et al., 2016) to encode the tweet and the group separately.”
DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inference (2018, unknown): “…a p-value of (Bowman et al., 2015) 83.9% 80.6% (Vendrov et al., 2015) 98.8% 81.4% (Mou et al., 2016) 83.3% 82.1% (Bowman et al., 2016) 89.2% 83.2% (Liu et al., 2016b) 84.5% 84.2% (Yu and Munkhdalai, 2017a) 86.2% 84.6% (Rockt¨aschel et al., 2015) 85.3% 83.5% (Wang and Jiang, 2016) 92.0% 86.1%…”
Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks (2018, background): “Liu et al. (2016b); Zhang et al. (2017b) proposed multi-task learning architecture with different methods of sharing information, and can be further enhanced with an external memory (Liu et al., 2016a) which is shared across the participant tasks.”
Deep Learning--based Text Classification (2020, background): “Liu et al. [82] present a LSTM model with inner-attention for NLI.”
Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference (2016, methodology): “…encoders [Mou et al., 2016] 3.5M 83.3 82.1 (5) 300D SPINN-PI encoders [Bowman et al., 2016] 3.7M 89.2 83.2 (6) 600D BiLSTM intra-attention encoders [Liu et al., 2016] 2.8M 84.5 84.2 (7) 300D NSE encoders [Munkhdalai and Yu, 2016a] 3.0M 86.2 84.6 (8) 100D LSTM with attention [Rocktäschel et al.,…”
Natural Language Inference with External Knowledge (2017, methodology): “Moreover, when we compose the (Vendrov et al., 2015) 81.4 Tree CNN (Mou et al., 2016) 82.1 SPINN-PI (Bowman et al., 2016) 83.2 NTI (Munkhdalai & Yu, 2016b) 83.4 Intra-Att BiLSTM (Liu et al., 2016) 84.2 Self-Att BiLSTM (Lin et al., 2017) 84.2 NSE (Munkhdalai & Yu, 2016a) 84.6 Gated-Att BiLSTM (Chen e”