A Decomposable Attention Model for Natural Language Inference

By Ankur P. Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit. 2016

1,424 citations recorded by the source corpus.

We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, neutral): “For applications involving text pairs, a common pattern is to independently encode text pairs before applying bidirectional cross attention, such as Parikh et al. (2016); Seo et al.”
Structured Attention Networks (2017, methodology): “Our baseline model/setup is essentially the same as that of Parikh et al. (2016).”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “This decomposition avoids the quadratic complexity that is trivially parallelizable (Parikh et al., 2016).”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “, 2015): Decomposable Attention (Parikh et al., 2016) and ESIM (Chen et al.”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, background): “jor benchmark for machine learning work on sentence understanding and spurring work on core representation learning techniques for NLU, such as attention (Wang and Jiang, 2016; Parikh et al., 2016), memory (Munkhdalai and Yu, 2017), and the use of parse structure (Mou et al.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “…fasttext word vectors (Mikolov et al., 2018) and learned character vectors, applies a shared bidirectional LSTM to both parts, applies co-attention (Parikh et al., 2016) to share information between the two parts, applies another bi-LSTM to both parts, pools the result, and uses the pooled…”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “We evaluate four existing models on this dataset: Decomposable Attention (DA): Introduced by Parikh et al. (2016), this model uses a form of attention to align words in the premise and in the hypothesis and to make predictions based on the aggregation of this alignment.”
Universal Transformers (2018, methodology): “The Transformer model in particular relies entirely on a self-attention mechanism (Parikh et al., 2016; Lin et al., 2017) to compute a series of context-informed vector-space representations of the symbols in its input and output, which are then used to predict distributions over subsequent symbols…”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “We adapted two neural entailment models, DecompAttn (Parikh et al., 2016) and DGEM (Khot et al., 2018), to the task of answering multiple-choice questions.”
Annotation Artifacts in Natural Language Inference Data (2018, methodology): “We ran this experiment on three high-performing NLI models: the De-composable Attention Model (DAM; Parikh et al., 2016), 5 the Enhanced Sequential Inference Model (ESIM; Chen et al., 2017), 6 and the Densely Interactive Inference Network (DIIN; Gong et al., 2018).”
Music Transformer: Generating Music with Long-Term Structure (2018, background): “Intuitively, self-attention (Parikh et al., 2016) could be a good match for this task.”
Self-Attention with Relative Position Representations (2018, methodology): “Attention-based models have therefore used position encodings or biased attention weights based on distance (Parikh et al., 2016).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “For example, we observe a 0.2-0.5% improvement over the decomposable attention model (Parikh et al., 2016).”
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, neutral): “On the other side, the previous work Parikh et al. [2016], Shaw et al. [2018], Huang et al. [2018], Dai et al. [2019], Yang et al. [2019], Raffel et al. [2020], Ke et al. [2020], He et al. [2020], Huang et al. [2020] focuses on relative position encoding, which typically encodes the relative…”
On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “Word embeddings have become the de facto feature representation across NLP (Parikh et al. 2016; Seo et al. 2017, for example).”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “Mitigating model limitations in NLI stress tests: Liu et al. (2019a) observed catastrophic forgetting when fine-tuning DA (Parikh et al., 2016) and ESIM (Chen et al., 2016) models trained on MultiNLI (Williams et al., 2018) with the numerical-reasoning and antonym stress tests released by Naik et…”