Word Representations: A Simple and General Method for Semi-Supervised Learning

By Joseph P. Turian, Lev-Arie Ratinov, Yoshua Bengio. 2010

2,346 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “Pretrained word embeddings are considered to be an integral part of modern NLP systems, offering significant improvements over embeddings learned from scratch (Turian et al., 2010).”
Deep Contextualized Word Representations (2018, background): “Due to their ability to capture syntactic and semantic information of words from large scale unlabeled text, pretrained word vectors (Turian et al., 2010; Mikolov et al., 2013; Pennington et al., 2014) are a standard component of most state-ofthe-art NLP architectures, including for question answeri”
context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “This was done in tasks, such as chunking, NER, semantic role labeling, and co-reference resolution (Turian et al., 2010; Collobert et al., 2011; Melamud et al., 2016), mostly by considering the embeddings of words in a window around the target of interest.”
GloVe: Global Vectors for Word Representation (2014, methodology): “We conclude that the GloVe vectors are useful in downstream NLP tasks, as was ﬁrst shown for neural vectors in (Turian et al., 2010).”
Distributed Representations of Sentences and Documents (2014, methodology): “…NLP applications such as word representation, named entity recognition, word sense disam-biguation, parsing, tagging and machine translation (Col-lobert & Weston, 2008; Turney & Pantel, 2010; Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011b; Huang et al., 2012; Zou et al., 2013).”
Experience Grounds Language (2020, background): “learned from text capture a great deal of information about meaning in and out of context (Collobert and Weston, 2008; Turian et al., 2010; Mikolov et al., 2013; McCann et al., 2017).”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “A selection of successful applications of this approach include sequence labeling (Turian et al., 2010), parsing (Chen and Manning, 2014), sentiment (Socher et al.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 26, 21, 18, 9], and some of the resulting word vectors were made available for future research and comparison2.”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “8 Word representations Pretrained word vectors (Turian et al., 2010; Pennington et al., 2014) have been an essential component in state-of-the-art NLP systems.”
Parameter-Efficient Transfer Learning for NLP (2019, background): “Turian et al. (2010) show that pre-trained embeddings of words outperform those trained from scratch.”
A Convolutional Neural Network for Modelling Sentences (2014, methodology): “As the dataset is rather small, we use lower-dimensional word vectors with d = 32 that are initialised with embeddings trained in an unsupervised way to predict contexts of occurrence (Turian et al., 2010).”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In language domain, several neural network models have been proposed to learn word/n-gram representations [28, 29, 30, 31, 32, 33], sentence representations [34] and paragraph/document representations [35].”
Natural Language Processing (Almost) from Scratch (2011, methodology): “atively small training set (RCV1, 37M words), unlikely to contain enough instances of the rare words. Secondly, they predict the correctness of the nal word of each window instead of the center word (Turian et al., 2010), eectively restricting the model to unidirectional prediction. Finally, they do”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, background): “Distributed representations of words (Rumelhart et al., 1988; Collobert et al., 2011; Turian et al., 2010; Huang et al., 2012; Mikolov et al., 2013; Pennington et al., 2014) have found wide applicability in a variety of NLP tasks.”
ERNIE: Enhanced Language Representation with Informative Entities (2019, methodology): “As these pre-trained word representations capture syntactic and semantic information in textual corpora, they are often used as input embeddings and initialization parameters for various NLP models, and offer significant improvements over random initialization parameters (Turian et al., 2010).”
End-to-end Neural Coreference Resolution (2017, background): “Word representations The word embeddings are a ﬁxed concatenation of 300-dimensional GloVe embeddings (Pennington et al., 2014) and 50-dimensional embeddings from Turian et al. (2010), both normalized to be unit vectors.”
Exploiting Similarities among Languages for Machine Translation (2013, background): “…and to various other NLP tasks such as word representation learning, named entity recognition, disambiguation, parsing, and tag-w ging (Collobert and Weston, 2008; Turian et al., 2010; Socher et al., 2011; Socher et al., 2013; Col-lobert et al., 2011; Huang et al., 2012; Mikolov et al., 2013a).”