XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

By Edoardo Maria Ponti, Goran Glavavs, Olga Majewska, Qianchu Liu, Ivan Vulic, Anna Korhonen. 2020

457 citations recorded by the source corpus.

In order to simulate human language capacity, natural language processing systems must complement the explicit information derived from raw text with the ability to reason about the possible causes and outcomes of everyday situations. Moreover, the acquired world knowledge should generalise to new languages, modulo cultural differences. Advances in machine commonsense reasoning and cross-lingual transfer depend on the availability of challenging evaluation benchmarks. Motivated by both demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We benchmark a range of state-of-the-art models on this novel dataset, revealing that current methods based on multilingual pretraining and zero-shot fine-tuning transfer suffer from the curse of multilinguality and fall short of performance in monolingual settings by a large margin. Finally, we propose ways to adapt these models to out-of-sample resource-lean languages where only a small corpus or a bilingual dictionary is available, and report substantial improvements over the random baseline. XCOPA is available at this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “For example, XTREME (Hu et al., 2020) introduces a benchmark covering 40 languages across multiple NLU and retrieval tasks, XCOPA (Ponti et al., 2020) is a commonsense reasoning dataset for eleven languages, and MLQA (Lewis et al., 2020b) is a dataset for extractive question answering across seven l”
PaLM 2 Technical Report (2023, methodology): “We compare to PaLM, GPT-4 (OpenAI, 2023b), and the state of the art (SOTA) for each dataset.5 We employ the instruction-tuned version of PaLM 2 (see Appendix A.2 for the detailed instruction tuning results) except for the multilingual XCOPA dataset.6 PaLM 2 outperforms PaLM across all datasets and a”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “For CCR, we rely on XCOPA (Ponti et al., 2020a), a crosslingual version of COPA (Roemmele et al.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “To better understand the multilingual reasoning abilities of large pretrained language models, we extend our experiments to two additional multilingual reasoning benchmarks, XCOPA (Ponti et al., 2020) and XL-WiC (Raganato et al., 2020).”
mGPT: Few-Shot Learners Go Multilingual (2022, unknown): “(Liang et al., 2020; Tikhonov and Ryabinin, 2021a; Ponti et al., 2020) with “zero-shot” and “few-shot” methods.”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “13 According to the received wisdom from text-only cross-lingual transfer (Ponti et al., 2020; Ruder et al., 2021), both these factor usually play a pivotal role in determining the downstream performance of each target language.”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “At each step, we select the language with the largest number of articles belonging to the language family and its genus that are least represented in the current sample of languages (Ponti et al., 2020); see Appendix for a full list.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “This phenomenon, termed the curse of multilinguality, can be attenuated by increasing the capacity of the model (Artetxe et al., 2020; Pfeiffer et al., 2020c; Chau et al., 2020) or through additional training for particular language pairs (Pfeiffer et al., 2020c; Ponti et al., 2020).”
XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, unknown): “The advantage of the translation-based approach adapted from Ponti et al. (2020) is twofold.”
Probing Pretrained Language Models for Lexical Semantics (2020, background): “…constraints: a) availability of comparable pretrained (language-specific) monolingual LMs; b) availability of evaluation data; and c) typological diversity of the sample, along the lines of recent initiatives in multilingual NLP (Gerz et al., 2018; Hu et al., 2020; Ponti et al., 2020, inter alia).”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, methodology): “Additionally, we evaluate AlexaTM 20B on multilingual tasks including XNLI (Conneau et al., 2018), XCOPA (Ponti et al., 2020), Paws-X (Yang et al., 2019), and XWinograd (Tikhonov & Ryabinin, 2021).”
MEGA: Multilingual Evaluation of Generative AI (2023, background): “For all languages, the monolingual setting outperforms the zero-shot cross-lingual setting, which follows the same trend the we observe with XNLI and XCOPA.”
Few-shot Learning with Multilingual Language Models (2021, methodology): “For both XNLI and XCOPA, using the English prompts on average yield significantly better performance than using the machine-translated prompts.”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, background): “Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP (Nekoto et al., 2020; Ponti et al., 2020; Ruder et al., 2021), data-to-text datasets are mostly limited to English and a small number of other languages.”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “and reasoning datasets (Ponti et al., 2020; Lin et al., 2021) altogether only cover around 30 languages.”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “Only translating/generating high-quality evaluation data is becoming an attractive middle ground option (Liu et al., 2021; Ponti et al., 2020; Yin et al., 2021) that at least allows us to judge the success of cross-lingual transfer in a culturally appropriate way.”