XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation

By Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao and 14 more. 2020

398 citations recorded by the source corpus.

In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

MLSUM: The Multilingual Summarization Corpus (2020, background): “XGLUE In order to train and evaluate their performance across a diverse set of cross-lingual tasks, Liang et al. (2020) recently released XGLUE, covering both Natural Language Understanding and Generation scenarios.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Two such projects, XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020) incorporate MLQA as part of their aggregated benchmark.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Publicly accessible large-scale multi-task benchmarks [12, 62, 61, 23, 37] have facilitated recent advances [16, 72, 42, 71, 64, 18, 47] in NLP.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “Following Wu et al. (2020), with 30% of probability we switch all the entity mentions (can be one or multiple) from the original language to their counterparts in the other language, ending up with sentences like “ Ο µ πά µ α later reﬂected on his years ...", where we substituted “Obama" with a…”
Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning (2020, methodology): “In the meantime, Conneau et al. (2019); Liang et al. (2020) constructed a better pre-trained model to improve the cross-lingual performance by enlarging the amount of pre-trained data and the size of the pre-trained model.”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “…leaderboards include GLUE and SuperGLUE for natural language understanding (Wang et al., 2018, 2019), XTREME and XGLUE (Hu et al., 2020; Liang et al., 2020) for multilingual understanding, the WMT shared tasks (Barrault et al., 2020) for machine translation, and GEM and GENIE for…”
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (2020, background): “Many of these multilingual resources have been bundled into cross-lingual benchmarks, such as XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020).”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “The pretraining corpus consists of a typologically weighted set of languages covered by cross-lingual benchmarks, such as XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020).”
MTG: A Benchmark Suite for Multilingual Text Generation (2021, background): “XGLUE (Liang et al., 2020) is a cross-lingual benchmark dataset for nine understanding tasks and two generation tasks.”
MEGA: Multilingual Evaluation of Generative AI (2023, methodology): “We would also like to include more standard NLP tasks, as well as tasks that are taken from real-world applications, such as the Bing tasks included in XGLUE (Liang et al., 2020).”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, background): “…8 from the web to obtain state-of-the-art performance via task-specific fine-tuning (Devlin et al., 2018; Pfeiffer et al., 2021; Li and Liang, 2021; Hu et al., 2021; Lester et al., 2021) on various benchmark datasets (Wang et al., 2018, 2019; Hu et al., 2020; Liang et al., 2020; Lu et al., 2021).”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, methodology): “…a large-scale multilingual corpus, new state-of-the-art results on XNLI, MLQA and NER can be obtained. mBART [11] and Unicoder described in XGLUE [12] extend the multilingual models to multilingual text generation tasks based on the encoder-decoder framework and use diﬀerent denoising…”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, result): “In addition, to make the comparison as fair as possible, we take Unicoder-VL as another baseline, as it employs the same pre-training data during image-language pre-training.”
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, methodology): “We also report the results for two new tasks MSNTG and MSQG introduced from GLGE (Liu et al., 2020a).”