From Paraphrase Database to Compositional Paraphrase Model and Back

By John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu. 2015

290 citations recorded by the source corpus.

The Paraphrase Database (PPDB; Ganitkevitch et al., 2013) is an extensive semantic resource, consisting of a list of phrase pairs with (heuristic) confidence estimates. However, it is still unclear how it can best be used, due to the heuristic nature of the confidences and its necessarily incomplete coverage. We propose models to leverage the phrase pairs from the PPDB to build parametric paraphrase models that score paraphrase pairs more accurately than the PPDB’s internal scores while simultaneously improving its coverage. They allow for learning phrase embeddings as well as improved word embeddings. Moreover, we introduce two new, manually annotated datasets to evaluate short-phrase paraphrasing models. Using our paraphrase model trained using PPDB, we achieve state-of-the-art results on standard word and bigram similarity tasks and beat strong baselines on our new short phrase paraphrase tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “9 Paragram Paraphrase Database (PPDB) fit word embeddings (Wieting et al., 2015) 63.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, methodology): “…questions, we apply it to our multiple-choice setting following the procedure by Kembhavi et al. (2017): Specifically, we score each answer-option by computing the similarity value of it’s output span with each of the candidate answers, computed by phrasal similarity tool of Wieting et al. (2015).”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “We tune over the strategy in our experiments.”
Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “Additional experiments with Word2Vec (Mikolov et al., 2013), Fasttext (Bojanowski et al., 2017), Paragram-SL-999 (Wieting et al., 2015) are in the appendix.”
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, positive): “We follow the training procedure of Wieting et al. (2015) and Wieting et al. (2016b), described below.”
Commonsense Knowledge Base Completion (2016, methodology): “We use the GloVe (Penning-ton et al., 2014) embeddings trained on 840 billion tokens of Common Crawl web text and the P ARAGRAM -SimLex embeddings of Wieting et al. (2015), which were tuned to have strong performance on the SimLex-999 task (Hill et al., 2015).”
Identifying Well-formed Natural Language Questions (2018, methodology): “, 2006; Bowman et al., 2015), paraphrasing (Wieting et al., 2015) etc. we rely on the wisdom of the crowd to get such annotations in order to make the data collection scalable and language-independent.”
Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks (2015, methodology): “We use Dim k = 25 -dimensional PARAGRAM vectors (Wieting et al., 2015) only on the MSRP task since they were developed for paraphrase tasks, having been trained on word pairs from the Para-phrase Database (Ganitkevitch et al., 2013).”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, methodology): “They generate sentence embeddings from diverse pre-trained word embeddings among which are unsupervised word embed-dings such as GloVe (Pennington et al., 2014) as well as supervised word embeddings such as paragram-SL999 (PSL) (Wieting et al., 2015) trained on the Paraphrase Database (Ganitkevitch ”
Paraphrasing Revisited with Neural Machine Translation (2017, methodology): “Wieting et al. (2015) take the paraphrases contained in PPDB and embed them into a low-dimensional space using a recursive neural network similar to Socher et al. (2013).”
Neural Belief Tracker: Data-Driven Dialogue State Tracking (2016, methodology): “Using GloVe vectors (Pennington et al., 2014) in place of Paragram-SL999 (Wieting et al., 2015) drastically reduced the models’ goal tracking capabilities.”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “l text like NMT systems and PPDB, can be used for learning embeddings for words and sentences. Several have used PPDB as a knowledge resource for training or improving embeddings (Faruqui et al.,2015;Wieting et al., 2015;Mrksiˇ ´c et al. ,2016). Several have used NMT architectures and training setti”
Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement (2016, methodology): “For the STS2014, WikiQA, and TrecQA experiments, we used 300-dimension PARAGRAM - SL 999 embeddings from Wieting et al. (2015) and the PARAGRAM - PHRASE embeddings from Wieting et al. (2016), trained on word pairs from the Paraphrase Database (PPDB) (Ganitkevitch et al., 2013).”
Question Answering as Global Reasoning Over Semantic Abstractions (2018, methodology): “ (as opposed to entailment) is taken to be the average of the entailment scores in both directions. (b) For longer phrasal alignments (e.g., when aligning phrasal verbs) we use the Paragram system of Wieting et al. (2015). The ﬁnal optimization is done on Eq. (1). The ﬁrst part of the objective is t”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, background): “Representing sentences by the average of their constituent word representations is surprisingly effective in encoding the semantic information of sentences (Wieting et al., 2015; Adi et al., 2016) and close to the state-of-the-art in these datasets.”
Informing Unsupervised Pretraining with External Linguistic Knowledge (2019, background): “It also further validates intuitions from relevant work on specialising static word embeddings (Wieting et al., 2015; Mrkˇsi´c et al., 2017) that steering distributional models towards capturing true semantic similarity (as also done here) has a positive impact on language understanding…”
UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement (2016, methodology): “We also ﬁnd that the PARAGRAM - PHRASE word embeddings are highly beneﬁcial, outperforming both GloVe word embeddings (Pennington et al., 2014) and the PARAGRAM - SL 999 embeddings of Wieting et al. (2015).”