Dependency-Based Word Embeddings

By Omer Levy, Yoav Goldberg. 2014

1,184 citations recorded by the source corpus.

While continuous word embeddings are gaining popularity, current models are based solely on linear contexts. In this work, we generalize the skip-gram model with negative sampling introduced by Mikolov et al. to include arbitrary contexts. In particular, we perform experiments with dependency-based contexts, and show that they produce markedly different embeddings. The dependencybased embeddings are less topical and exhibit more functional similarity than the original skip-gram embeddings.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “The analysis presented in Levy and Goldberg (2014b) is valid for every co-occurrence matrix that describes the joint distribution of two random variables.”
nocaps: novel object captioning at scale (2019, methodology): “We extend the original model – which incorporated GloVe [31] and dependency embeddings [21] – to incorporate the recently proposed ELMo [32] model, which increased performance in our preliminary experiments.”
Identifying and Reducing Gender Bias in Word-Level Language Models (2019, background): “On the other hand, a larger window size captures topicality (Levy and Goldberg, 2014).”
Deep Biaffine Attention for Neural Dependency Parsing (2016, methodology): “We use a larger but more thoroughly regularized parser than other recent BiLSTM-based approaches, with biaffine classifiers to predict arcs and labels.”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “10 by Levy and Goldberg (2014) trained on the English Wikipedia, three different GloVe embeddings11 from Pennington et al. (2014) trained either on Wikipedia 2014 + Gigaword 5 (about 6 billion tokens) or on Common Crawl (about 840 billion tokens), and the Komninos and Manandhar (2016) embeddings12…”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, positive): “Levy and Goldberg (2014c) made a similar observation, and hypothesized that this is a result of the increasing number of zero-cells, which may cause SVD to prefer a factorization that is very close to the zero matrix.”
Linguistic Regularities in Sparse and Explicit Word Representations (2014, background): “Another popular choice of context is the syntactic relations the word participates in (Lin, 1998; Pad´o and Lapata, 2007; Levy and Goldberg, 2014).”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, background): “Recently, word embedding techniques have been studied for their mathematical properties (Levy and Goldberg, 2014b; Stratos et al., 2015), yielding a better understanding of the underlying optimization criteria.”
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model (2015, methodology): “We follow the method introduced by Levy and Goldberg (2014b), which returns the word x maximizing cos( king,x ) × cos( woman,x ) cos( man,x ) .”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “The grouping is also syntactic, grouping together words that share an inflection (Levy & Goldberg, 2014a).”
Do Neural Language Representations Learn Physical Commonsense? (2019, neutral): “We also use Dependency Based Word Embeddings (Levy & Goldberg, 2014), as they may more directly capture the relations between objects and their affordances.”
That’s So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets (2015, background): “Frame-Semantic Embeddings Although lexical (Mikolov et al., 2013a) and dependency based embeddings (Levy and Goldberg, 2014) have been studied, semantic-based embedding is still less understood.”
nocaps : novel object captioning at scale (2019, methodology): “We report results for three models, the baseline model (UpDown), the baseline model using Glove [9] and dependency-based [7] word embeddings (UpDown + GD) and our ELMo-based model (UpDown + ELMo +CBS).”
Sentiment Embeddings with Applications to Sentiment Analysis (2016, neutral): “We propose learning sentiment embeddings that encode sentiment of texts in continuous word representation.”
Semantic Structure and Interpretability of Word Embeddings (2017, methodology): “NLP tasks using word embeddings, the individual embedding dimensions are commonly considered to be uninterpretable [15].”
Capturing Semantic Similarity for Entity Linking with Convolutional Neural Networks (2016, methodology): “For learning the vectors, we use the standard word2vec toolkit (Mikolov et al., 2013) with vector length set to 300, window set to 21 (larger windows produce more semantically-focused vectors (Levy and Goldberg, 2014)), 10 negative samples and 10 iterations through Wikipedia.”
Community Interaction and Conflict on the Web (2018, methodology): “We trained the embeddings using 100 passes over the full 40 months of Reddit posts, with stochastic gradient descent code adapted from Levy et al. [48].”