Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext

By John Wieting, Jonathan Mallinson, Kevin Gimpel. 2017

111 citations recorded by the source corpus.

We consider the problem of learning general-purpose, paraphrastic sentence embeddings in the setting of Wieting et al. (2016b). We use neural machine translation to generate sentential paraphrases via back-translation of bilingual sentence pairs. We evaluate the paraphrase pairs by their ability to serve as training data for learning paraphrastic sentence embeddings. We find that the data quality is stronger than prior work based on bitext and on par with manually-written English paraphrase pairs, with the advantage that our approach can scale up to generate large training sets for many languages and domains. We experiment with several language pairs and data sources, and develop a variety of data filtering techniques. In the process, we explore how neural machine translation output differs from human-written sentences, finding clear differences in length, the amount of repetition, and the use of rare words.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Machine Translation

Citations in the corpus

An efficient framework for learning sentence representations (2018, neutral): “Wieting et al. (2017) explore the use of machine translation to obtain more paraphrase data via back-translation and use it for learning paraphrastic embeddings.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “Additionally, we found that even after training on these corpora, the model would struggle with paraphrases, so we also translated SNLI sentences from English to German and back using the Nematus machine translation system [81, 73].”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (2018, methodology): “Since no large-scale dataset of sentential para-phrases exists publicly, we follow Wieting et al. (2017) and automatically generate millions of paraphrase pairs using neural backtranslation .”
Style Transfer Through Back-Translation (2018, methodology): “More recently, Mallinson et al. (2017) and Wieting et al. (2017) showed how neural back-translation can be used to generate paraphrases.”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “nce similarity scores in semantic evaluations. They used pairs of NMT systems, one to translate an English sentence into multiple foreign translations and the other to then translate back to English. Wieting et al. (2017) only uses the NMT system to generate training data for training sentence embed”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “…NMT model as a generative paraphraser.3 Semantic Similarity Parallel corpora in many language pairs have been used to produce fixed-size, multilingual sentence representations (Schwenk and Douze, 2017; Wieting et al., 2017; Artetxe and Schwenk, 2018; Wieting et al., 2019; Raganato et al., 2019).”
Image Pivoting for Learning Multilingual Multimodal Representations (2017, unknown): “In the semantic textual similarity task (STS), we use the textual embeddings from our model to compute the similarity between a pair of sen-Model (Wieting et al., 2017) − 83.7 84.5 85.0 MLMME (Calixto et al., 2017) VGG19 − 72 .”
Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition (2019, background): “There has been a line of attempts to learn compositional phrase representations (e.g., Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Wieting et al., 2017; Poliak et al., 2017), but many of these are tailored to a specific type of phrase or to a fixed number of constituent words, and they a”
Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “…3 for details) by ranking them with 3 techniques: semantic similarity using the PARAGRAM - PHRASE model from (Wieting et al., 2016b), trigram-overlap, and translations score (the latter two were both used in (Wieting et al., 2017) Table 4 shows the results of the different ﬁlter-ing methods.”
Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021, background): “Wieting et al. (2017); Wieting and Gimpel (2018) use translation models to generate paraphrases via back-translation.”
On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (2018, background): “Many researchers suggest that state-of-the-art NMT models learn representations that capture the meaning of sentences (Gu et al., 2016; Johnson et al., 2017; Zhou et al., 2017; Andreas and Klein, 2017; Neubig, 2017; Koehn, 2017).”
Effective Data Augmentation Approaches to End-to-End Task-Oriented Dialogue (2019, methodology): “For the sentence-level paraphrasing, we use a seq2seq paraphrase model which contains a bidirectional LSTM encoder and LSTM decoder together with an attention network.1 The model is trained on a mixed data set consisting of paraphrases from para-nmt-5m, Quora question pairs, SNLI and Semeval [11], [”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, background): “Examples include bilingual pivoting over phrases (Callison-Burch et al., 2006; Ganitkevitch et al., 2013), and automatic translation of one side of the bitext (Wieting et al., 2017; Wieting and Gimpel, 2018; Hu et al., 2019).”
Demystifying Prompts in Language Models via Perplexity Estimation (2022, methodology): “Step 2: Paraphrasing using backtranslation Our second step takes as input the paraphrases from GPT3 (in addition to the seed set of prompts) and translates them into different languages and back into English to get additional prompt paraphrases (Wieting et al., 2017).”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, background): “This perturbation randomly substitutes some words in an English text with their WordNet (Miller, 1998)”