Improving Word Representations via Global Context and Multiple Word Prototypes

By Eric Hsin-Chun Huang, Richard Socher, Christopher D. Manning, Andrew Y. Ng. 2012

1,277 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “In an unsupervised setting, such representations were found useful for measuring context-sensitive similarity (Huang et al., 2012), word sense disambiguation (Chen et al., 2014), word sense induction (K˚ageb¨ack et al., 2015), lexical substitution (Melamud et al., 2015b), sentence completion (Liu…”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, background): “These vectors can be trained in an unsupervised fashion to capture distributional similarities (Collobert and Weston, 2008; Huang et al., 2012) but then also be ﬁne-tuned and trained to speciﬁc tasks such as sentiment detection (Socher et al., 2011b).”
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (2018, methodology): “To our knowledge, the Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) is the only existing benchmark that speciﬁcally focuses on the dynamic nature of word semantics.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 26, 21, 18, 9], and some of the resulting word vectors were made available for future research and comparison2.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In language domain, several neural network models have been proposed to learn word/n-gram representations [29, 30, 31, 32, 33, 34], sentence representations [35] and paragraph/document representations [36].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Note that more important activity and object words are higher up in this tree structure. supervised model of Huang et al. (2012) which can learn single word vector representations from both local and global contexts.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, background): “All training images x ( i ) ∈ X y of a seen class y ∈ Y s are mapped to the word vector w y corresponding to the class name.”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, background): “Access to document-level features can improve both classical language models (Mikolov & Zweig, 2012) and word embeddings (Huang et al., 2012).”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, background): “Distributed representations of words (Rumelhart et al., 1988; Collobert et al., 2011; Turian et al., 2010; Huang et al., 2012; Mikolov et al., 2013; Pennington et al., 2014) have found wide applicability in a variety of NLP tasks.”
Exploiting Similarities among Languages for Machine Translation (2013, background): “…and to various other NLP tasks such as word representation learning, named entity recognition, disambiguation, parsing, and tag-w ging (Collobert and Weston, 2008; Turian et al., 2010; Socher et al., 2011; Socher et al., 2013; Col-lobert et al., 2011; Huang et al., 2012; Mikolov et al., 2013a).”
Semantic Parsing via Paraphrasing (2014, methodology): “We also ex-periment with publicly released word embeddings (Huang et al., 2012), which were trained using both local and global context.”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, methodology): “12 Huang et al. Huang et al. (2012) present a NLM that learns word embeddings to maximise the likelihood of predicting the last word in a sentence s based on (i) the previous words in that sentence (lo-cal context - as with Collobert and Weston (2008)) and (ii) the document d in which that word occu”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “We probe representations from the resulting English models with the Word in Context (WiC; Pilehvar and Camacho-Collados, 2019), Stanford Contextual Word Similarity (SCWS; Huang et al., 2012), and the syntactic evaluation (Marvin and Linzen, 2018) datasets.”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, neutral): “(2) The Wikipedia corpus used in Huang et al. (2012) (train/test: 1,104,937/100,000 and V :20,000).”
Bayesian Neural Word Embedding (2016, methodology): “The word similarity evaluation includes several different datasets: WordSim353 [16], SCWS [19], Rare Words (RW) [20], MEN [21] and Sim-Lex999 [22].”
Deep Learning: Methods and Applications (2014, background): “[169] recognized the limitation of the earlier work on word embeddings in that these models were built with only local context and one representation per word.”
Transductive Multi-View Zero-Shot Learning (2015, methodology): “[31] first employed a linguistic model [15] as the intermediate semantic representation.”
Recurrent Convolutional Neural Networks for Text Classification (2015, methodology): “…(Bengio, Courville, and Vincent 2013) have led to new ideas for solving the data sparsity problem, and many neural models for learning word representations have been proposed (Bengio et al. 2003; Mnih and Hinton 2007; Mikolov 2012; Collobert et al. 2011; Huang et al. 2012; Mikolov et al. 2013).”