Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

By John Wieting, Kevin Gimpel. 2017

78 citations recorded by the source corpus.

We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of Wieting et al. (2016b). While they found LSTM recurrent networks to underperform word averaging, we present several developments that together produce the opposite conclusion. These include training on sentence pairs rather than phrase pairs, averaging states to represent sequences, and regularizing aggressively. These improve LSTMs in both transfer learning and supervised settings. We also introduce a new recurrent architecture, the Gated Recurrent Averaging Network, that is inspired by averaging and LSTMs while outperforming them both. We analyze our learned models, finding evidence of preferences for particular parts of speech and dependency relations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “Wieting et al. (2015) and Wieting & Gimpel (2017) use paraphrase data to learn an encoder that maps synonymous phrases to similar embeddings using a margin loss.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “This method is known as hypercolumns (Hariharan et al., 2015) in CV and is used by Peters et al. (2017), Peters et al. (2018), Wieting and Gimpel (2017), Conneau et al. (2017), and McCann et al. (2017) who use language modeling, paraphrasing, entailment, and Machine Translation (MT) respectively…”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “…level similarity and semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et al. (2016a,b); He et al.…”
A Survey of Data Augmentation Approaches for NLP (2021, background): “, 2016) × 3 Depends Input 3 SCPN (Wieting and Gimpel, 2017) × 3 const Input 3 SEMANTIC TEXT EXCHANGE (Feng et al.”
Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, background): “While there are various ways of scoring semantic similarity between pairs of texts based on embeddings (Le and Mikolov, 2014; Wieting and Gimpel, 2017), they do not explicitly penalize unnatural sentences, and generating sentences requires surrounding context (Le and Mikolov, 2014) or training a sep”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “Our use of an intermediate dataset leads to the best results, but this may be due to our efforts in optimizing learning for this setting (Wieting et al., 2016b; Wieting and Gimpel, 2017).”
Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext (2017, methodology): “So we select models and a loss function from prior work (Wieting et al., 2016b; Wieting and Gimpel, 2017).”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, background): “Much recent work has investigated how to encode the semantic meaning of a sentence into a fixed high dimensional embedding space (Kiros et al., 2015; Wieting and Gimpel, 2017).”
Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “model on SimpWiki, a dataset of aligned sentences from Simple English and standard English Wikipedia (Coster and Kauchak,2011). It has been shown useful for training sentence embeddings in past work (Wieting and Gimpel, 2017). However, Table8shows that training on PARANMT leads to gains in correlati”
GenAug: Data Augmentation for Finetuning Text Generators (2020, methodology): “(5)We also tried syntactic paraphrasing using SCPNs (Wieting and Gimpel, 2017) but found the paraphrase quality poor and hard to control for meaning preservation and fluency.”
Zero-training Sentence Embedding via Orthogonal Basis (2018, background): “Wieting & Gimpel (2017) propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia.”
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (2019, background): “Wieting and Gimpel (2017) The White House’s head, he described the conversation as ridiculous.”
Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training (2019, background): “, 2017), Recurrent Neural Networks (RNNs) (Yang et al., 2016; Wieting and Gimpel, 2017), Convolutional Neural Networks (CNNs) (Kim, 2014), self-attention blocks (Devlin et al.”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “ing only words or character n-grams) as well the LSTM model, BILSTM, from Wieting and Gimpel (2017). These models use a contrastive loss with a margin.”