XNLI: Evaluating Cross-lingual Sentence Representations

By Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, Veselin Stoyanov. 2018

1,635 citations recorded by the source corpus.

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “, 2019a), and XNLI (Conneau et al., 2018), and matches the state-of-the-art on SQuAD.”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “, 2019; Conneau et al., 2019)1. CC25 includes languages from different families and with varied amounts of text (Table 1). Following Lample and Conneau (2019), we rebalanced the corpus by up/down-sampling text from each language i with a ratio λi:”
Cross-lingual Language Model Pretraining (2019, methodology): “Recent developments in learning and evaluating cross-lingual sentence representations in many languages [12] aim at mitigating the English-centric bias and suggest that it is possible to build universal cross-lingual encoders that can encode any sentence into a shared embedding space.”
Unifying Question Answering and Text Classification via Span Extraction (2019, background): “…al., 2018; Howard and Ruder, 2018; Peters et al., 2018; McCann et al., 2017) have been shown to transfer remarkably well on down-stream tasks including text classiﬁcation, question answering, machine translation, and summarization (Wang et al., 2018; Rajpurkar et al., 2016; Conneau et al., 2018).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…in natural language inference, where the dataset SNLI (Bowman et al., 2015) was extended to cover more genres (Williams et al., 2018) and languages (Conneau et al., 2018) and subsequently has been used to explore whether adversarial data augmentation techniques are useful for evaluation (Nie et…”
Learning Compact Metrics for MT (2021, neutral): “Evidence suggests that training on many languages improves performance on languages for which there is little training data, including the zero-shot setup, in which no ﬁne-tuning data is available (Conneau and Lample, 2019; Sellam et al., 2020b; Conneau et al., 2018; Pires et al., 2019).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “We evaluate only XNLI and skip TyDiQAGoldP in this setting, as it is unreasonable to expect a generative model that was fine-tuned to always copy spans from the context to spontaneously “undo” corruptions and predict novel spans.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Following XLM-R (Conneau et al., 2018), we increase the vocabulary size to 250 , 000 wordpieces.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “However, previous works reported a signiﬁcant performance gap be-tween English and the target language, e.g. for classiﬁcation (Conneau et al., 2018) and Question Answering (Lewis et al., 2019) tasks.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Outside of QA, XNLI (Conneau et al., 2018) has gained popularity for natural language understanding.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, neutral): “It contains data in 15 languages (Conneau et al., 2018).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “We evaluate each model on the XNLI (Conneau et al., 2018) corpus by using the training data in each language.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Given recent progress in cross-lingual tasks such as document classiﬁcation (Lewis et al., 2004; Klementiev et al., 2012; Schwenk and Li, 2018), semantic role labelling (Akbik et al., 2015) and NLI (Conneau et al., 2018), we argue that while multilingual QA training data might be useful but not stri”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Following Conneau et al. (2019), we select the best single model on the joint dev set of all the languages.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “XNLI: We also evaluate mSLAM models on the XNLI sentence-pair classification task (Conneau et al., 2018) to understand its strengths and weaknesses on text understanding tasks.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “Multilingually pretrained NLP models such as mBERT [2], XLM-R [8] or mT5 [9] brought significant improvements in multilingual language understanding [10, 11].”
The ROOTS Search Tool: Data Transparency for LLMs (2023, methodology): “Our tool enables sampled checks of benchmark data, and was already successfully used to ﬁnd 14 that BLOOM should not be evaluated on XNLI (Conneau et al., 2018).”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “We believe the non-translation nature of OCNLI makes it more suitable than XNLI (Conneau et al., 2018) as an NLU task specific for Chinese.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “Lample and Conneau (2019) propose translation language modeling (TLM) as a way to leverage parallel data and obtain a new state of the art on the cross-lingual natural language inference (XNLI) benchmark (Conneau et al., 2018).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “…work has noted potential issues with standard datasets: MLDoc (Schwenk & Li, 2018) can be solved by matching keywords (Artetxe et al., 2019), while MultiNLI, the dataset from which XNLI (Conneau et al., 2018b) was derived, contains superﬁcial cues that can be exploited (Gururangan et al., 2018).”