Notes on Noise Contrastive Estimation and Negative Sampling

By Chris Dyer. 2014

109 citations recorded by the source corpus.

Estimating the parameters of probabilistic models of language such as maxent models and probabilistic neural models is computationally difficult since it involves evaluating partition functions by summing over an entire vocabulary, which may be millions of word types in size. Two closely related strategies---noise contrastive estimation (Mnih and Teh, 2012; Mnih and Kavukcuoglu, 2013; Vaswani et al., 2013) and negative sampling (Mikolov et al., 2012; Goldberg and Levy, 2014)---have emerged as popular solutions to this computational problem, but some confusion remains as to which is more appropriate and when. This document explicates their relationships to each other and to other estimation techniques. The analysis shows that, although they are superficially similar, NCE is a general parameter estimation technique that is asymptotically unbiased, while negative sampling is best understood as a family of binary classification models that are useful for learning word representations but not as a general-purpose estimator.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Compositional Fairness Constraints for Graph Embeddings (2019, methodology): “…of these scoring functions using contrastive learning methods that make use of a corruption distribution such as noise contrastive estimation (Dyer, 2014; Mnih & Teh, 2012) and similar variants (Bose et al., 2018), where the loss over a batch of edges E batch ⊆ E train is given by: e ∈E edge…”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, background): “Previous attempts have been made to explain negative sampling (Dyer, 2014) using the idea of Noise Contrastive Estimation (NCE) (Gutmann & Hyv ¨ arinen, 2010) which relies on the assumption that the data distribution belongs to some known parametric family.”
Intra-agent speech permits zero-shot task acquisition (2022, methodology): “To understand if this term represents a liability for the approach and to generate an algorithm with potentially complementary strengths, we also developed a contrastive, energy-based approach, amounting to multi-class classification, to approximate it [17–19].”
Adversarial Contrastive Estimation (2018, methodology): “With respect to our ACE framework, the difference between NCE and Negative Sampling is inconsequential, so we continue the discussion using NCE.”
Neural information retrieval: at the end of the early years (2017, background): “tailed discussion of NCE and NEG, see notes by Dyer [60].”