ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

By John Wieting, Kevin Gimpel. 2017

245 citations recorded by the source corpus.

We describe ParaNMT-50M, a dataset of more than 50 million English-English sentential paraphrase pairs. We generated the pairs automatically by using neural machine translation to translate the non-English side of a large parallel corpus, following Wieting et al. (2017). Our hope is that ParaNMT-50M can be a valuable resource for paraphrase generation and can provide a rich source of semantic knowledge to improve downstream natural language understanding tasks. To show its utility, we use ParaNMT-50M to train paraphrastic sentence embeddings that outperform all supervised systems on every SemEval semantic textual similarity competition, in addition to showing how it can be used for paraphrase generation.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “Our sentence embedding model is optimized for similarity search, trained with a triplet loss on ground-truth paraphrases, parallel sentences as well as as hard negatives (Wieting et al., 2016; Wieting and Gimpel, 2017).”
Editing Factual Knowledge in Language Models (2021, methodology): “To this end, we generate semantically equivalent inputs using round-trip translation (Sennrich et al., 2016; Wieting and Gimpel, 2018).”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (2018, methodology): “In this paper, we use the PARANMT-50M corpus from Wieting and Gimpel (2017).”
Post-hoc Interpretability for Neural NLP: A Survey (2021, unknown): “, 2019) on existing counterfactual datasets (Kaushik et al., 2020; Gardner et al., 2020; Zhang et al., 2019; Sakaguchi et al., 2020; Wieting and Gimpel, 2018; McCoy et al., 2019).”
Paraphrase Generation: A Survey of the State of the Art (2021, background): “Twitter URL Twitter URL (Lan et al., 2017) is constructed by collecting large-scale sentential paraphrases from Twitter by linking tweets through ParaNMT ParaNMT (Wieting and Gimpel, 2018) is a dataset of more than 50 million English-English sentential paraphrase pairs.”
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020, background): “Para-phrase (Wieting and Gimpel, 2018) and semantic relatedness (Marelli et al., 2014) tasks are such examples.”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “…Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et al., 2015), labelled entailment corpora (Conneau et al., 2017), image-caption data (Kiela et al., 2017), raw text labelled with discourse relations (Nie et al., 2017), or parallel corpora (Wieting & Gimpel, 2017).”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, methodology): “For baselines, we compare against Lag VAE (He et al., 2019), CGMH (Miao et al., 2019), UPSA (Liu et al., 2020), BT (Wieting and Gimpel, 2018), Corruption (Hegde and Patil, 2020), Con-RPG (Meng et al., 2021), and MCPG (Chen et al., 2022b).”
Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021, methodology): “For the training data, we consider ParaNMT-50M (Wieting and Gimpel, 2018), a paraphrase dataset containing over 50 million pairs of reference sentences and the corresponding paraphrases as well as the quality scores.”
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (2019, methodology): “Table 1: Comparison of the semantic similarity model used in this paper (SIM) with a number of strong baselines including the model of (Wieting and Gimpel, 2018) and the top 3 performing STS systems for each year.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “[139] train a sentence encoder, д, on a set of paraphrase pairs (from ParaNMT corpus [140]) using the max margin loss: l (s, s ′) = max (0,δ − cos (д(s ),д(s ′)) + cos (д(s ),д(t ))), where δ is the margin, s and s ′ are paraphrases, and t is a negative example obtained by random sampling the other ”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, background): “The current trend of using machine translation systems to produce augmented datasets for machine translation itself (Sennrich et al., 2016) and for monolingual tasks like classiﬁ-cation (Yu et al., 2018) and paraphrasing (Wieting and Gimpel, 2018) is a good example of this.”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “We promote output diversity by training fpara on an aggressivelyfiltered subset of PARANMT-50M (Wieting and Gimpel, 2018), a large corpus of backtranslated text.”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, background): “Underlined: current unsupervised SoTA on test set (Wieting and Gimpel, 2018).”
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2019, methodology): “Despite their inability to understand word order, they are surprisingly effective on sentence similarity tasks ( Arora et al., 2017; Wieting and Gimpel, 2018; Ethayarajh, 2018) (b) Complex contextualized sentence encoders, such as Long Short Term Memory Networks (LSTM) (Hochreiter and Schmidhuber,…”