Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference

By Qian Chen, Xiao-Dan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen. 2017

100 citations recorded by the source corpus.

The RepEval 2017 Shared Task aims to evaluate natural language understanding models for sentence representation, in which a sentence is represented as a fixed-length vector with neural networks and the quality of the representation is tested with a natural language inference task. This paper describes our system (alpha) that is ranked among the top in the Shared Task, on both the in-domain test set (obtaining a 74.9% accuracy) and on the cross-domain test set (also attaining a 74.9% accuracy), demonstrating that the model generalizes well to the cross-domain data. Our model is equipped with intra-sentence gated-attention composition which helps achieve a better performance. In addition to submitting our model to the Shared Task, we have also tested it on the Stanford Natural Language Inference (SNLI) dataset. We obtain an accuracy of 85.5%, which is the best reported result on SNLI when cross-sentence attention is not allowed, the same condition enforced in RepEval 2017.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “MNLI The Multi-Genre Natural Language Inference Corpus (Williams et al., 2018) is a crowd-sourced collection of sentence pairs with textual entailment annotations.”
Stress Test Evaluation for Natural Language Inference (2018, background): “Within the scope of these benchmark datasets, state-of-the-art deep learning-based sentence encoder models 2 (Nie and Bansal (2017), Chen et al. (2017), Conneau et al. (2017), Balazs et al. (2017), inter alia ) have been shown to consistently achieve high accuracies, which may lead us to believe…”
Natural Language Inference over Interaction Space (2017, background): “The exact match feature is simple while found extremely useful in reading comprehension task (Chen et al., 2017a).”
The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations (2017, methodology): “h performing models, does not use within-sentence attention. The best performing system overall is an ensemble by Chen et al., which is based closely on the Enhanced Sequential Inference Model (ESIM; Chen et al., 2017a) but with attention only within each sentence, rather than between the two. Looki”
EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference (2019, background): “(2010); Clark (2018) argue for a systematic knowledge-oriented approach in NLI by evaluating specific semantic analysis tasks, identifying quantitative reasoning in particular as a focus area.”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, methodology): “It has been applied to various NLP tasks, e.g., question answering (Wang et al., 2017), neural machine translation (Bahdanau et al., 2015), sentiment analysis (Qian et al., 2017), natural language inference (Liu et al., 2016), etc.”
A Compare-Propagate Architecture with Alignment Factorization for Natural Language Inference (2017, methodology): “While this term was coined by (Wang and Jiang, 2016a), many prior NLI models follow this design (Wang et al., 2017; Parikh et al., 2016; Gong et al., 2017; Chen et al., 2017a).”
Enhancing Sentence Embedding with Generalized Pooling (2018, methodology): “In Cross CBOW (Williams et al., 2017) 64.8 64.5 BiLSTM (Williams et al., 2017) 66.9 66.9 BiLSTM gated-pooling (Chen et al., 2017b) 73.5 73.6 Shortcut stacked BiLSTM (Nie and Bansal, 2017) (Lin et al., 2017) 61.99 77.30 CNN max pooling (Lin et al., 2017) 62.05 78.15 BiLSTM self-attention (Lin et…”
Dynamic Integration of Background Knowledge in Neural NLU Systems (2017, methodology): “We hypothesize that the attention mechanisms within ESIM jointly with powerful, pre-trained word representations allow for the recovery of some important lexical relations when trained on a large dataset.”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “It has been successfully applied in models aimed to solve various tasks of NLP, like machine translation[25, 33, 42] and natural language inference task[6, 21].”
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification (2019, background): “…to train complicated neural models which have achieved the state-of-the-art results (Bowman et al., 2015; Parikh et al., 2016; Sha et al., 2016; Chen et al., 2017b,c; Munkhdalai and Yu, 2017; Nie and Bansal, 2017; Conneau et al., 2017; Gong et al., 2018; Tay et al., 2018; Ghaeini et al., 2018).”
Deep Learning for Entity Matching: A Design Space Exploration (2018, methodology): “For example, recent work [7, 49] proposed using Bi-LSTMs—a state-of-the-art RNN—with attention to learn representation vectors for sentence pairs.”
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information (2018, methodology): “…joint methods, which make up for the lack of interaction in the former methods, use cross-features as an attention mechanism to express the word-or phrase-level alignments for performance improvements (Wang, Hamza, and Florian 2017; Chen et al. 2017b; Gong, Luo, and Zhang 2018; Yang et al. 2016).”
Distance-based Self-Attention Network for Natural Language Inference (2017, methodology): “In Chen et al. (2017) and Ni and Bansal (2017), short-cut connections in stacked LSTM, in combination with max-pooling originally suggested by Conneau et al. (2017), were proven effective in improving performance, recording the state-of-the-art performance in MultiNLI.”
Exploring End-to-End Differentiable Natural Logic Modeling (2020, methodology): “A system that The right-most column of Table 4 shows that while ESIM and BERT achieve very high development accuracy on the upward data, they fail to generalize to the downward monotone test set.”
Convolutional Interaction Network for Natural Language Inference (2018, methodology): “Rockt¨aschel et al. (2015) ﬁrstly use LSTM with attention for text matching task.”
Learning to Compose Task-Specific Tree Structures (2017, methodology): “600D (300+300) Gated-Attention BiLSTM (Chen et al. 2017) 85.”