Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

By Marco Baroni, Georgiana Dinu, Germán Kruszewski. 2014

1,522 citations recorded by the source corpus.

Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GloVe: Global Vectors for Word Representation (2014, background): “Currently, prediction-based models garner substantial support; for example, Baroni et al. (2014) argue that these models perform better across a range of tasks.”
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021, background): “A neural LM sees the prompt as a sequence of continuous word vectors (Baroni et al., 2014).”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, methodology): “16 In contrast to recent results that emphasize the superiority of NLMs over alternatives (Baroni et al., 2014), we observed no clear advantage for the NLM over the VSM or LSA when considering the association-based gold standards WS-353 and MEN together.”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “Intrinsic evaluation tests the semantic relationships between words [3, 32], and, thus leverages word embeddings only.”
Word Representations via Gaussian Embedding (2014, background): “A comparison to all of the state of the art word-embedding numbers for different dimensionalities as in (Baroni et al., 2014) is out of the scope of this evaluation.”
Recurrent Convolutional Neural Networks for Text Classification (2015, methodology): “In this work, we use the Skip-gram model to pre-train the word embedding. this model is the state-of-the-art in many NLP tasks (Baroni, Dinu, and Kruszewski 2014).”
SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability (2015, methodology): “They use WordNet (Miller, 1995), Mikolov Embeddings (Mikolov et al., 2013; Baroni et al., 2014) and PPDB (Ganitkevitch et al., 2013).”
Linguistic Regularities in Sparse and Explicit Word Representations (2014, neutral): “…shows that ﬁnding analogies through vector arithmetic is actually a form of balancing word similarities, and that, contrary to the recent ﬁndings of Baroni et al. (2014), under certain conditions traditional word similarities induced by explicit representations can perform just as well as neural…”
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model (2015, methodology): “The recent evaluation of Baroni et al. (2014b) suggests that the C-BOW model introduced by Mikolov et al. (2013a) is, consistently, the best across many tasks.”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “Here we follow the setting and the proposed al-gorithm in (Baroni et al., 2014; Schnabel et al., 2015) – we cluster words (via their representations) using the classical k -Means algorithm (with ﬁxed k ).”
Emerging Language Spaces Learned From Massively Multilingual Corpora (2018, background): “Predictive models based on neural network classifiers and neural language models [8,9] have superseded models that are purely based on co-occurrence counts (see [10] for a comparison of common approaches).”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, background): “They’re just a few points below the state-of-the-art MEN correlation (0.8), achieved by Baroni et al. (2014) with a corpus 3 larger than ours and extensive tuning.”
Identifying Customer Needs from User-Generated Content (2019, background): “High-quality word embeddings can be pre-trained on a large text corpus and then applied in downstream NLP tasks (Baroni et al., 2014).”