Linguistic Regularities in Continuous Space Word Representations

By Tomas Mikolov, Wen-tau Yih, Geoffry Zweig. 2013

3,957 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “Alternatively, we considered an objective function similar to the negative sampling approach of Mikolov et al. (2013b).”
GloVe: Global Vectors for Word Representation (2014, methodology): “We conduct experiments on the word analogy task of Mikolov et al. (2013a), a variety of word similarity tasks, as described in (Luong et al., 2013), and on the CoNLL-2003 shared benchmark Word analogies.”
Distributed Representations of Sentences and Documents (2014, methodology): “Our technique is inspired by the recent work in learning vector representations of words using neural networks (Bengio et al., 2006; Collobert & Weston, 2008; Mnih & Hinton, 2008; Turian et al., 2010; Mikolov et al., 2013a;c).”
Distributed Representations of Words and Phrases and their Compositionality (2013, background): “The follow up work includes applications to automatic speech recognition and machine translation [14, 7], and a wide range of NLP tasks [2, 20, 15, 3, 18, 19, 9].”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “Popular methods for learning word representations include (Collobert et al., 2011; Mikolov et al., 2013c; Mnih and Kavukcuoglu, 2013; Pennington et al., 2014): all are based on unsupervised objec-4 tives of predicting words or word frequencies from raw text.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector(”King”) - vector(”Man”) + vector(”Woman”) results in a vector that is closest to the vector representation of the word Queen [20].”
Findings of the 2016 Conference on Machine Translation (2016, methodology): “The word embeddings were learned by using the Continuous Bag-of-Words (CBOW) model (Mikolov et al., 2013), trained on the Google’s billion-word corpus,18 with a vocabulary size of 527K words.”
Generating Sentences by Editing Prototypes (2017, methodology): “ich randomly samples an edit vector z^ ˘p(z), instead using ^zderived from f(x 1;x 2). We also compare our accuracies to the simpler task of solving the word, rather than sentence-level analogies in (Mikolov et al., 2013a) using GloVe. This task is substantially simpler, since the goal is to identif”
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “Second, the vector differences between words in embeddings have been shown to represent relationships between words [32, 26].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “We show that using a linear sentence encoder, linguistic regularities [12] also carry over to multimodal vector spaces.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Single word vector spaces are widely used (Turney and Pantel, 2010) and successful at classifying single words and capturing their meaning (Collobert and Weston, 2008; Huang et al., 2012; Mikolov et al., 2013).”
DeepWalk: online learning of social representations (2014, methodology): “These neural language models have been used to capture the semantic and syntactic structure of human language [6], and even logical analogies [28].”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “Word embeddings capture features such as number and tense (Mikolov et al., 2013), so aggregated word embeddings will naturally track these features’ majority values in a sentence.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “The internal representations of a neural network have been found to capture certain aspects of meaning, such as semantic similarity (Mikolov et al., 2013; Clark, 2015).”
Learning Gender-Neutral Word Embeddings (2018, methodology): “…embeddings serve as a fundamental building block for a broad range of NLP applications (dos Santos and Gatti, 2014; Bahdanau et al., 2014; Zeng et al., 2015) and various approaches (Mikolov et al., 2013b; Pen-nington et al., 2014; Levy et al., 2015) have been proposed for training the word vectors.”
Character-Aware Neural Language Models (2015, methodology): “Using the CharCNN and highway layers for representation learning (e.g. as input into word2vec (Mikolov et al. 2013)) remains an avenue for future work.”
Learning to Generate Reviews and Discovering Sentiment (2017, background): “Mikolov et al. (2013) showed that word-level recurrent language modelling supports the learning of useful word vectors and we are interested in pushing this line of work.”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, background): “However, not only are these techniques intractable to train with high-order context vectors, they also lack the neural network’s ability to semantically generalize (Mikolov et al., 2013) and learn non-linear relationships.”