PAWS: Paraphrase Adversaries from Word Scrambling

By Jason Baldridge, Luheng He. 2019

613 citations recorded by the source corpus.

Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York. This paper introduces PAWS (Paraphrase Adversaries from Word Scrambling), a new dataset with 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State-of-the-art models trained on existing datasets have dismal performance on PAWS (<40% accuracy); however, including PAWS training data for these models improves their accuracy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual information fail even with PAWS training examples. As such, PAWS provides an effective instrument for driving further progress on models that better exploit structure, context, and pairwise comparisons.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “…estimates for human performance on several tasks that we did not ultimately include in our benchmark, including GAP (Webster et al., 2018), PAWS (Zhang et al., 2019), Quora Insincere Questions, 8 Ultraﬁne Entity Typing (Choi et al., 2018b), and Empathetic Reactions datasets (Buechel et al.,…”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Paws Wiki (Zhang et al., 2019) • Coreference resolution tests the ability to identify expressions of the same entity in some given text.”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “We provide an easily-extendable set of perturbation macros, which capture three common types of perturbations in prior work, shown in Table 3: First, syntactic rewriting primarily involves shu ﬄ ing text to create paraphrases (Zhang et al., 2019) or adversarial examples (Iyyer et al., 2018).”
A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Zhang et al. (2019b) Explore backtranslation and feature discrimination for DA.”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “HANS (McCoy et al., 2019), a challenge set for NLI, contains 10k pairs of premises and hypotheses created based on 10 heavily fallible syntactic templates, and therefore compensates rarer structural changes that may be missed by PAWS.”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “PAWS includes two parts PAWS QQP , which is based on the QQP data, and PAWS Wiki .”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…qa Lin et al. 2020a onestop_english cls/other Vajjala and Lučić 2018 openbookqa qa/multiple-choice qa Mihaylov et al. 2018 paws cls/paraphrase Zhang et al. 2019 piqa other Bisk et al. 2020 poem_sentiment cls/sentiment analysis Sheng and Uthus 2020 proto_qa other Boratko et al. 2020 qa_srl…”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “A subset of the PAWS dev and test sets (Zhang et al., 2019) was translated to six other languages by professional translators and is used for evaluation, while the PAWS training set is used for training.”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “Stateof-the-art solutions utilize self-attentive Transformer blocks (Liu, 2019; Liu and Lapata, 2019; Zhang et al., 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including…”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “PAWS is a dataset that contains pairs of sentences with a high lexical overlap (Zhang et al., 2019).”
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (2019, methodology): “First, human translation does not require high-quality multilingual part-of-speech tag-gers or named entity recognizers, which play a key role in the data generation process used in Zhang et al. (2019).”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, methodology): “PAWS (Zhang et al., 2019): A dataset of altered Wikipedia sentences using word swapping and back-translation.”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, methodology): “XNLI (Conneau et al., 2018), NER (Sang, 2002; Sang and De Meulder, 2003), POS Tagging (Kim et al., 2017), MLQA (Lewis et al., 2019b) and PAWS-X (Yang et al., 2019a) are 5 multilingual datasets built for specific tasks.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “In natural language processing (NLP), a current focus is on challenge datasets that are carefully crafted to test particular aspects of models, e.g., HANS (McCoy et al., 2019), PAWS (Zhang et al., 2019), and CheckList (Ribeiro et al., 2020).”
Language Models are General-Purpose Interfaces (2022, methodology): “• Paraphrase Detection is to detect the semantic similarity of two sentences: QQP (Wang et al., 2018), MRPC (Dolan and Brockett, 2005), Paws Wiki (Zhang et al., 2019).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, methodology): “Further, PAWS-X (Yang et al., 2019b) evaluates paraphrase identiﬁ-cation over 6 languages.”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “On Quora Question Pairs (QQP) (Iyer et al., 2017), one of the largest PI dataset, Zhang et al. (2019) overlap but different meanings through word swap-ping and back-translation.”