Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

By John Wieting, Kevin Gimpel. 2017

144 citations recorded by the source corpus.

We extend the work of Wieting et al. (2017), back-translating a large parallel corpus to produce a dataset of more than 51 million English-English sentential paraphrase pairs in a dataset we call ParaNMT-50M. We find this corpus to be cover many domains and styles of text, in addition to being rich in paraphrases with different sentence structure, and we release it to the community. To show its utility, we use it to train paraphrastic sentence embeddings using only minor changes to the framework of Wieting et al. (2016b). The resulting embeddings outperform all supervised systems on every SemEval semantic textual similarity (STS) competition, and are a significant improvement in capturing paraphrastic similarity over all other sentence embeddings We also show that our embeddings perform competitively on general sentence embedding tasks. We release the corpus, pretrained sentence embeddings, and code to generate them. We believe the corpus can be a valuable resource for automatic paraphrase generation and can provide a rich source of semantic information to improve downstream natural language understanding tasks.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Generating Datasets with Pretrained Language Models (2021, background): “Often, labeled datasets such as paraphrase databases (Wieting and Gimpel, 2018) or natural language inference datasets (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019) are used for supervised learning.”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (2018, methodology): “In this paper, we use the P ARA NMT-50M corpus from Wieting and Gimpel (2017).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, neutral): “For instance, we find that the lexical overlap (F1 measured between two bags of words) for the entailment pairs (SNLI + MNLI) is 39%, while they are 60% and 55% for QQP and ParaNMT.”
Negative Training for Neural Dialogue Response Generation (2019, methodology): “Towards that end, we train a seq2seq paraphrase model using the paraNMT data-set (Wieting and Gimpel, 2017), with a model of the same structure as described in Section 2.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “An alternative is to start with much more plentiful bitext and back-translate one side into the language of the other to create synthetic paraphrases on which to train (Prakash et al., 2016; Wieting and Gimpel, 2018; Hu et al., 2019a,b,c).”
Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models (2018, background): “To obtain a ceiling performance of the model, we also conducted a separate study where Turkers were presented with 100 randomly sampled ground-truth ParaNMT-5M paraphrase pairs.”
Zero-training Sentence Embedding via Orthogonal Basis (2018, background): “Wieting & Gimpel (2017) propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia.”
TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation (2018, methodology): “We concatenate pre-trained Glove (Pennington et al., 2014) and para-phrase (Wieting and Gimpel, 2017) embeddings.”
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2020, background): “But MT paraphrasing systems cycletranslate through a pivot language (Federmann et al., 2019; Wieting and Gimpel, 2018), which requires supervised bilingual translation data, with an implicit notion of interlingual paraphrasing.”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “…Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et al., 2015), labelled entailment corpora (Conneau et al., 2017), image-caption data (Kiela et al., 2017), raw text labelled with discourse relations (Nie et al., 2017), or parallel corpora (Wieting & Gimpel, 2017).”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, methodology): “Other methods, which we will call in-training, encourage the alignment of embeddings during training (e.g., Bollegala et al. (2015); Yang et al. (2017)).”
A Review on Language Models as Knowledge Bases (2022, methodology): “External memory has previously been used for neural networks (Bahdanau et al., 2015; Weston et al., 2015; Graves et al., 2014; Ahn et al., 2017).”
Effective Data Augmentation Approaches to End-to-End Task-Oriented Dialogue (2019, methodology): “1 The model is trained on a mixed data set consisting of paraphrases from para-nmt-5m, Quora question pairs, SNLI and Semeval [11], [12].”
Text Detoxification using Large Pre-trained Neural Models (2021, methodology): “phrase dataset (Wieting and Gimpel, 2018) with our toxicity classifier described in Section 5.”
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2019, methodology): “We create naive sentence meta-embeddings by concatenating (Yin and Sch¨utze, 2016) or averaging (Coates and Bollegala, 2018) length-normalized sentence embeddings (where ˆ x = x / || x || 2 ): Yin and Sch¨utze (2016) use Singular Value Decomposition (SVD) to compactify concatenated word embedding…”
Transfer Fine-Tuning: A BERT Case Study (2019, methodology): “Due to studies on sentential paraphrase collection (Lan et al., 2017) and generation (Wieting and Gimpel, 2018), a million-scale paraphrase corpus is ready for use.”
QuickEdit: Editing Text & Translations via Simple Delete Actions (2017, methodology): “ll as the output matrix have 256 dimensions. For WMT'14 en-de and WMT'14 de-en, both encoders and decoders have 15 layers (9 layers with 512 hidden units, 4 1 Posterior to our experiments, (Wieting and Gimpel, 2017) released an even large dataset that might be used in our setting. 2 http:/”