Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model

By Ngoc-Quan Pham, Germán Kruszewski, Angeliki Lazaridou, Marco Baroni. 2015

48 citations recorded by the source corpus.

We introduce C-PHRASE, a distributional semantic model that learns word representations by optimizing context prediction for phrases at all levels in a syntactic tree, from single words to full sentences. C-PHRASE outperforms the state-of-theart C-BOW model on a variety of lexical tasks. Moreover, since C-PHRASE word vectors are induced through a compositional learning objective (modeling the contexts of words combined into phrases), when they are summed, they produce sentence representations that rival those generated by ad-hoc compositional models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “…semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et al. (2016a,b); He et al. (2015); Pham et al. (2015)).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “Our ﬁrst striking result is the good overall performance of Bag-of-Vectors, conﬁrming early insights that aggregated word embeddings capture surprising amounts of sentence information (Pham et al., 2015; Arora et al., 2017; Adi et al., 2017).”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “dings, we ﬁnd that informationbearing content words, such as poverty, kidding, humanitarian, 18, and july have the largest L 2 norms, while words such as of, it, to, hereby and the have the smallest. Pham et al. (2015) noted this same phenomenon in their closely-related compositional model. Interest”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “, 2017), PV-DBOW (Lau and Baldwin, 2016), C-PHRASE (Kruszewski et al., 2015), ECNU (Tian et al.”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, methodology): “This observation can be attributed to the fact that a big chunk of the data that the C-PHRASE model is trained on comes from English Wikipedia, helping it to perform well on datasets involving deﬁnition and news items.”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, background): “is likely more important when training the model on sentences since they have so much more space to have diverse meaning unlike prior work on learning on text snippets (Wieting et al., 2015, 2016b,a; Pham et al., 2015). 6.5 Simple Models We ﬁrst investigate how well simple models (Word Avg., Trigram”
Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext (2017, background): “g general purpose sentence embeddings has used autoencoders (Socher et al.,2011;Hill et al.,2016), encoder-decoder architectures (Kiros et al.,2015), or other learning frameworks (Le and Mikolov,2014;Pham et al., 2015).Wieting et al.(2016b) andHill et al.(2016) provide many empirical comparisons to ”
Dependency Based Embeddings for Sentence Classification Tasks (2016, methodology): “of previous research has explored methods for directly taking syntax into account for word embedding learning (Pham et al., 2015; Cheng and Kartsaklis, 2015; Hashimoto et al., 2014).”
Learning Visually Grounded Sentence Representations (2017, background): “Recent work has ranged from trying to learn to compose word embeddings (Le and Mikolov, 2014; Pham et al., 2015; Wieting et al., 2016; Arora et al., 2017), to neural architectures for predicting the previous and next sentences (Kiros et al., 2015) or learning representations via large-scale…”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, methodology): “In this paper, we propose a method that trains highly performant sentence embeddings (Pham et al., 2015; Hill et al., 2016; Pagliardini et al., 2017; McCann et al., 2017; Conneau et al., 2017) directly on bitext, obviating these intermediate steps and avoiding the noise and error propagation from…”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “in their use of both training data and training objectives. Methods include autoencoders (Socher et al., 2011; Hill et al., 2016) and other learning frameworks using raw text (Le & Mikolov, 2014; Pham et al., 2015; Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et ”
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (2019, unknown): “We are talking about the 21st century: servants et al., 2015; Wieting et al., 2016) have also found that when training semantic embeddings using an averaging function, embeddings that bear the most information regarding the meaning have larger norms.”
Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, background): “is likely more important when training the model on sentences since they have so much more space to have diverse meaning unlike prior work on learning on text snippets (Wieting et al., 2015, 2016b,a; Pham et al., 2015). 6.5 Simple Models We ﬁrst investigate how well simple models (Word Avg., Trigram”