The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

By Nikita Nangia, Adina Williams, Angeliki Lazaridou, Samuel R. Bowman. 2017

94 citations recorded by the source corpus.

This paper presents the results of the RepEval 2017 Shared Task, which evaluated neural network sentence representation learning models on the Multi-Genre Natural Language Inference corpus (MultiNLI) recently introduced by Williams et al. (2017). All of the five participating teams beat the bidirectional LSTM (BiLSTM) and continuous bag of words baselines reported in Williams et al. The best single model used stacked BiLSTMs with residual connections to extract sentence features and reached 74.5% accuracy on the genre-matched test set. Surprisingly, the results of the competition were fairly consistent across the genre-matched and genre-mismatched test sets, and across subsets of the test data representing a variety of linguistic phenomena, suggesting that all of the submitted systems learned reasonably domain-independent representations for sentence meaning.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “nd mismatched sets as competitions that will be open indeﬁnitely; Evaluations on a subset of the test set have previously been conducted with different leaderboards through the RepEval 2017 Workshop (Nangia et al., 2017). Thecorpusisavailable intwoformats—tabseparated text and JSON Lines (jsonl), fo”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “MNLI Multi-Genre Natural Language Inference is a large-scale, crowd-sourced entailment classi-ﬁcation task (Nangia et al., 2017 of sentences (i.e., a premise-hypothesis pair), the goal is to predict whether the hypothesis is an entailment , contradiction , or neutral with respect to the premise.”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “When these NLI datasets are constructed to facilitate the training and evaluation of natural language understanding (NLU) systems (Nangia et al., 2017), it is tempting to claim that systems achieving high accuracy on such datasets have successfully “understood” natural language or at least a…”
Stress Test Evaluation for Natural Language Inference (2018, methodology): “result in the RepEval-2017 shared task (Nangia et al., 2017).”
Natural Language Inference over Interaction Space (2017, neutral): “Unlike the observation from Nangia et al. (2017), we ﬁnd the out-of-domain test performance is consistently lower than in-domain test performance.”
Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces (2018, methodology): “We use the Multi-Genre NLI corpus ( MultiNLI ) from the RepEval 2017 shared task (Nangia et al., 2017).”
Grounded Textual Entailment (2018, methodology): “Linguistic phenomena Following the error analysis approach described in recent work (Nangia et al., 2017; Williams et al., 2018), we compiled a new list of linguistic features that can be of interest when contrasting SNLI and SNLI hard , as well as for evaluating RTE models.”
Distance-based Self-Attention Network for Natural Language Inference (2017, result): “Compared with the result of RepEVAL 2017 (Nangia et al., 2017), we can see that the Distance-based Self-Attention Network performs well.”
SemEval-2018 Task 12: The Argument Reasoning Comprehension Task (2018, methodology): “GIST uses pretrained word2vec embeddings as well as the ESIM model (Chen et al., 2017), trained on the SNLI (Bowman et al., 2015) and MultiNLI (Nangia et al., 2017) datasets.”
Semantics-aware BERT for Language Understanding (2019, background): “…on 4 diverse datasets, including Stanford Natural Language Inference (SNLI) (Bow-man et al. 2015), Multi-Genre Natural Language Inference (MNLI) (Nangia et al. 2017), Question Natural Language Inference (QNLI) (Rajpurkar et al. 2016) and Recognizing Textual Entailment (RTE) (Bentivogli et al.…”
On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (2018, methodology): “Their model signiﬁcantly outperforms the majority baseline and our results on Multi-NLI, SPR, and FN+.”
A question-entailment approach to question answering (2019, background): “More recently, the multiNLI corpus [51] was shared in the scope of the RepEval 2017 shared task2 [37].”
Universal Multimodal Representation for Language Understanding (2023, methodology): “For MNLI and QNLI, the possible reason for the marginal improvements would be that both of the datasets are quite large.”