Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

By Zhuang Ma, Michael Collins. 2018

180 citations recorded by the source corpus.

Noise Contrastive Estimation (NCE) is a powerful parameter estimation method for log-linear models, which avoids calculation of the partition function or its derivatives at each training step, a computationally demanding step in many cases. It is closely related to negative sampling methods, now widely used in NLP. This paper considers NCE-based estimation of conditional models. Conditional models are frequently encountered in practice; however there has not been a rigorous theoretical analysis of NCE in this setting, and we will argue there are subtle but important questions when generalizing NCE to the conditional case. In particular, we analyze two variants of NCE for conditional models: one based on a classification objective, the other based on a ranking objective. We show that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method; we analyze the statistical efficiency of the ranking-based and classification-based variants of NCE; finally we describe experiments on synthetic data and language modeling showing the effectiveness and tradeoffs of both methods.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Residual Energy-Based Models for Text Generation (2020, methodology): “With the theoretical guarantee of NCE, we can show that the optimum of the above objective is reached at data distribution with inﬁnite amount of data and model with enough capacity, which is also proved in Ma & Collins (2018) 2 .”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “The best results with local DIM were obtained using a mutual information bound based on NoiseContrastive Estimation (NCE – [Gutmann and Hyvärinen, 2010]), as used in various NLP applications [Ma and Collins, 2018], and applied to infomax objectives by van den Oord et al.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “� Furthermore, Equation (3) is closely connected with the ranking NCE loss [33, 45], a popular objective for training energy-based models.”
Dual Contradistinctive Generative Autoencoder (2020, positive): “Remark 1 (From [49, 54]) The following objective is minimized, i.e., the optimal critic h is achieved, when h ( f θ ( z ) , x ) = log p ( x | z ) + c ( x ) where c ( x ) is any function that does not depend on z .”
On Mutual Information Maximization for Representation Learning (2019, background): “Indeed, Ma and Collins (2018) show that I NCE is consistent for any number of negative samples 9 , and Poole et al. (2019) show that the signal-to-noise ratio increases with the number of negative samples.”
On Variational Bounds of Mutual Information (2019, background): “This provides a proof 2 that INCE is a lower bound on MI. Unlike INWJ where the optimal critic depends on both the conditional and marginal densities, the optimal critic for INCE is f(x, y) = log p(y|x) + c(y) where c(y) is any function that depends on y but not x (Ma & Collins, 2018).”
Residual Energy-Based Models for Text (2021, methodology): “In particular, we adopt the conditional noise contrastive estimation (NCE) objective (Ma and Collins, 2018; Gutmann and Hyvärinen, 2010) to our residual model energy function and then sample from the joint model using importance sampling.”
How to Train Your Energy-Based Model for Regression (2020, methodology): “Moreover, NCE has been applied to train EBMs for supervised classification tasks within language modeling [22, 32, 33, 35], where the target space Y is a large but finite set of possible labels.”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “Framed under the name negative sampling [17], contrastive techniques have been established as indispensable tools in scaling up the learning of intractable statistical models such as graphs [18, 19].”
Pre-Training Transformers as Energy-Based Cloze Models (2020, methodology): “Instead, we use (conditional) Noise-Contrastive Estimation (NCE) (Gutmann and Hyv¨arinen, 2010; Ma and Collins, 2018), which provides a way of efﬁciently training an un-normalized model that does not compute Z θ ( x \ t ) .”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Contrastive representation learning methods [17, 46, 53, 54, 61, 77, 84, 86, 87, 122, 124, 129] take as input pairs of positive and negative examples, and learn representations so that positive pairs have similar representations and negative pairs have dissimilar representations.”
Multi-label Contrastive Predictive Coding (2020, background): “This can be explained from a mutual information estimation perspective that CPC estimates are upper bounded by logm, so increasing m could reduce bias when the actual mutual information is much higher [35].”
Autoregressive Reasoning over Chains of Facts with Transformers (2020, methodology): “…interactions in the gradient, we also use the conditional ranking variant of Noise-Contrastive Estimation (NCE), which covers > 2 facts at once (Ma and Collins, 2018; Gutmann and Hyv¨arinen, 2010): Where x 1 is positive and x > 1 are negative, B is the batch size, and P n is a negative…”
Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021, unknown): “We use noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010; Ma and Collins, 2018) to jointly train the energy model.”
Energy-Based Models for Text (2020, methodology): “In particular, we adopt the conditional noise contrastive estimation (NCE) objective (Ma and Collins, 2018; Gutmann and Hyvärinen, 2010) to our residual model energy function and then sample from the joint model using importance sampling.”