Enhanced LSTM for Natural Language Inference

By Qian Chen, Xiao-Dan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen. 2016

1,185 citations recorded by the source corpus.

Reasoning and inference are central to human and artificial intelligence. Modeling inference in human language is very challenging. With the availability of large annotated data (Bowman et al., 2015), it has recently become feasible to train neural network based inference models, which have shown to be very effective. In this paper, we present a new state-of-the-art result, achieving the accuracy of 88.6% on the Stanford Natural Language Inference Dataset. Unlike the previous top models that use very complicated network architectures, we first demonstrate that carefully designing sequential inference models based on chain LSTMs can outperform all previous models. Based on this, we further show that by explicitly considering recursive architectures in both local inference modeling and inference composition, we achieve additional improvement. Particularly, incorporating syntactic parsing information contributes to our best result—it further improves the performance even when added to the already very strong model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, background): “3 Enhanced BiLSTM Inference Model (Chen et al., 2016) 87 .”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Our model is a re-implementation of the ESIM sequence model (Chen et al., 2017).”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “12.0% that ESIM+ELMo answered incorrectly, for each extracting both the gold ending and the model’s preferred ending.”
Deep Contextualized Word Representations (2018, methodology): “Our baseline, the ESIM sequence model from Chen et al. (2017), uses a biL-STM to encode the premise and hypothesis, followed by a matrix attention layer, a local inference layer, another biLSTM inference composition layer, and ﬁnally a pooling operation before the output layer.”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “For instance, while ESIM + ELMo obtained 59% accuracy on SWAG, it obtains only 33.3% accuracy on HellaSwag .”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “The Enhanced Sequential Inference Model (ESIM; Chen et al., 2017) uses a modiﬁed version of a bidirectional LSTM to encode sentences.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “For Question Match and ESIM we also experiment with ELMo (Peters et al., 2018) which improved their score on Test with 0.4% and 1.8%.”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “Our model is the ESIM model (Chen et al., 2017), modiﬁed as needed to support regression tasks in addition to classiﬁcation.”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “e ESIM We use ESIM, a strong NLI model (Chen et al., 2016).”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, methodology): “In the evidence selection phase, participants measure the similarity between the claim and an evidence sentence candidate by training a classi-ﬁcation model like Enhanced LSTM (Chen et al., 2016) in a supervised setting or using string similarity function like TFIDF without trainable parameters.”
Transformers as Soft Reasoners over Language (2020, methodology): “We show that two other systems, BERT and ESIM (an LSTM-based model) [Chen et al. , 2017], are also able to learn these tasks, albeit with lower scores (95% and 80% respectively, vs. 98%).”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “We formulate the answer recycling problem as a constrained optimization based on the relevance and entailment scores between each candidate negative answer and the gold answer, as measured by state-of-the-art natural language inference models [10, 57, 15].”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, methodology): “We use the Enhanced Sequential Inference (Chen et al., 2017, ESIM) and Decompos-able Attention (Parikh et al., 2016, DA) models with GloVe embeddings (Pennington et al., 2014).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “the creation of many learned metrics, some of which use regression or deep learning (Stanojevic and Sima’an, 2014; Ma et al., 2017; Shimanaka et al., 2018; Chen et al., 2017; Mathur et al., 2019).”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, methodology): “, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”