PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

By Yinfei Yang, Yuan Zhang, Chris Tar, Jason Baldridge. 2019

432 citations recorded by the source corpus.

Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to capture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23% over the next best model. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge to drive multilingual research that better captures structure and contextual information.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “In this section we test this hypothesis on three wordlevel benchmarks: (i) transliteration, (ii) grapheme- Small Base Large XL XXL mT5 ByT5 mT5 ByT5 mT5 ByT5 mT5 ByT5 mT5 ByT5 In-language multitask (models fine-tuned on gold data in all target languages) WikiAnn NER 86.4 90.6 88.2 91.6 89.7 91.8 91.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “In the zero-shot setting, we use a batch size of 2 20 for XNLI and 2 16 for PAWS-X, NER, XQuAD, MLQA, and TyDi QA.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, neutral): “Our results, as a whole, demonstrate that ALIGN, a state-of-the-art multimodal dual encoder, is improved by adding a bitext ranking objective (Yang et al., 2019a) zero-shot image-text retrieval performance on well-resourced languages, and it dramatically improves performance on under-resourced…”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “PAWS-X The Cross-lingual Paraphrase Adversaries from Word Scrambling (Yang et al., 2019) dataset requires to determine whether two sentences are paraphrases.”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “We describe the models that we compare ( § 3.1), the experimental setting ( § 3.2), and the results on three classiﬁcation datasets: XNLI ( § 3.3), MLDoc ( § 3.4) and PAWS-X ( § 3.5).”
Pre-training via Paraphrasing (2020, methodology): “Variations on masked language models (MLMs) [Devlin et al., 2019, Liu et al., 2019, Yang et al., 2019b, Conneau et al., 2019, Lewis et al., 2019a, Raffel et al., 2019, Clark et al., 2020] provide highly effective self supervision for pre-training by removing and then reconstructing parts of an…”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, methodology): “XNLI (Conneau et al., 2018), NER (Sang, 2002; Sang and De Meulder, 2003), POS Tagging (Kim et al., 2017), MLQA (Lewis et al., 2019b) and PAWS-X (Yang et al., 2019a) are 5 multilingual datasets built for specific tasks.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, methodology): “These methods are similar to the prediction problems that BERT (De-vlin et al., 2019) and XLNet (Yang et al., 2019b) perform in their pre-training stages respectively.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “…cross-lingual natu-ral language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual question answering on MLQA (Lewis et al., 2020), XQuAD (Artetxe et al., 2020), and TyDiQA-GoldP (Clark et al.,…”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “Datasets that cover multiple languages for other natural understanding tasks, such as language inference (Conneau et al., 2018), question answering (Lewis et al., 2019; Artetxe et al., 2020a; Clark et al., 2020), and paraphrase identiﬁcation (Yang et al., 2019b) have received increasing attention.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Similar to previous approaches to create multilingual corpora, Yang et al. (2019a) used machine translation to create the training set for each target language in PAWS-X from the English training set in PAWS.”
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding (2020, background): “…multilingual datasets range over tasks such as document classiﬁcation, natural language inference, information retrieval, paraphrase identiﬁcation, and summarization (Klementiev, Titov, and Bhattarai 2012; Cer et al. 2017; Conneau et al. 2018; Sasaki et al. 2018; Yang et al. 2019; Zhu et al. 2019).”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “, 2018a), XWINO multilingual Winograd Schema challenge (Tikhonov and Ryabinin, 2021b), PAWSX adversarial paraphrases (Yang et al., 2019), Twitter HateSpeech detection (Davidson et al.”
A Comprehensive Overview of Large Language Models (2023, neutral): “Although decoder-only architectures have become a trend for LLMs, many recently proposed approaches [15], [110] use mode-switching tokens in text with encoder-decoder architectures to enable task-specific modes.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, background): “, 2021)), anaphora resolution (XWinograd (Tikhonov and Ryabinin, 2021)), and paraphrasing (PAWS-X (Yang et al., 2019)).”
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020, methodology): “The PAWS-X dataset (Yang et al., 2019) is a challenging paraphrase identiﬁcation dataset in six languages including Korean.”