GloVe: Global Vectors for Word Representation
By Jeffrey Pennington, Richard Socher, Christopher D. Manning. 2014
34,790 citations recorded by the source corpus.
Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Yejin Choi
- Graham Neubig
- Quoc V. Le
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Percy Liang
- Richard Socher
- Noah A. Smith
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Sebastian Ruder
- Trevor Darrell
- Ruslan Salakhutdinov
- Dacheng Tao
- Andrea Vedaldi
- Jianfeng Gao
- Ivan Vulic
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) methods.”
- A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “To apply this sequence-to-sequence approach to natural language, word embeddings (Mikolov et al., 2013; Pennington et al., 2014) are used to convert language tokens to vectors that can be used as inputs for these networks.”
- A Structured Self-attentive Sentence Embedding (2017, methodology): “We use 300 dimensional GloVe (Pennington et al., 2014) word embedding to initialize word embeddings.”
- Structured Attention Networks (2017, methodology): “The input representations are obtained by a linear transformation of the 300 -dimensional pretrained GloVe embeddings (Pennington et al., 2014) after normalizing the GloVe embeddings to have unit norm.”
- Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “For the word embedding model, we adopt the popular glove embeddings (Pennington et al., 2014) which are widely used in deep learning-based NLP domain.”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “Glove embedding (Pennington, Socher, and Manning 2014) represented as a d -dim embedding by combining the features/embedding described above.”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Word embeddings are initialized with GloVe vectors (Pennington et al., 2014) and fine-tuned during training.”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “To do so, we ran a series of controlled trials by swapping out pre-trained GloVe vectors (Pennington et al., 2014) for contextualized word vectors from each biLM computed by applying the learned weighted average ELMo pooling from Peters et al.”
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “We consider three different types of word representations: 300d GloVe vectors from Common Crawl (Pennington et al., 2014), 300d Numberbatch vectors retrofitted using ConceptNet relations (Speer et al., 2017), and 1024d ELMo contextual representations that show improvement on a variety of NLP tasks, i”
- Neural Network Acceptability Judgments (2018, methodology): “each layer j in an LSTM LM, though we depart from the original paper by using only a forward LM. (iii) In a more exploratory experiment, we also use pretrained 300- dimensional (6B) GloVe embeddings (Pennington et al., 2014). Note that these embeddings are trained on orders of magnitude more words t”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “(2017), the model uses a BiLSTM with temporal maxpooling and 300-dimensional GloVe word embeddings (Pennington et al., 2014) trained on 840B Common Crawl.”
- Deep Contextualized Word Representations (2018, methodology): “Unlike most widely used word embeddings (Pennington et al., 2014), ELMo word representations are functions of the entire input sentence, as described in this section.”
- An efficient framework for learning sentence representations (2018, methodology): “We use GloVe vectors (Pennington et al., 2014) as pre-trained word embeddings.”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “The word embedding is fixed during training and initialized from the p 1 = 300 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are fixed during training.”
- Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “The GloVe 300 dimensional word vectors released by Pennington et al. (2014) are used for word embeddings.”
- Learned in Translation: Contextualized Word Vectors (2017, methodology): “The MT-LSTM is used to compute a sequence of hidden states h = MT-LSTM ( GloVe ( w x )) .”
- SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “We use 50-dimensional GloVe word embeddings (Pennington et al., 2014) trained on a combination of Gigaword 5 (Parker et al.”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Distributed representations of words (or word embeddings) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) have shown to provide useful features for various tasks in natural language processing and computer vision.”
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “All three models are initialized with 300D reference GloVe vectors (840B token version; Pennington et al., 2014).”