Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

By Michael U Gutmann, Aapo Hyvärinen. 2010

2,742 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Adversarial Nets (2014, methodology): “Noise-contrastive estimation (NCE) [13] involves training a generative model by learning the weights that make the model useful for discriminating data from a fixed noise distribution.”
Pylearn2: a machine learning research library (2013, methodology): “Other costs implemented in the library include dropout [27] , contrastive divergence [26], noise contrastive estimation [24], score matching [28], denoising s core matching [52], softmax log likelihood for classification with MLPs, and Gaussian log likelihood fo r regression with MLPs.”
Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “es to push up the energies associated with x’s sampled from P(x)). Sampling for the negative phase is usually done by MCMC, although some unsupervised learning algorithms (Collobert and Weston, 2008; Gutmann and Hyvarinen, 2010; Bordes et al., 2013) involve “negative examples” that are sampled throu”
Residual Energy-Based Models for Text Generation (2020, methodology): “With the theoretical guarantee of NCE, we can show that the optimum of the above objective is reached at data distribution with inﬁnite amount of data and model with enough capacity, which is also proved in Ma & Collins (2018) 2 .”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To train g to predict future shot embeddings, we rely on noise-contrastive estimation [20].”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “The best results with local DIM were obtained using a mutual information bound based on NoiseContrastive Estimation (NCE – [Gutmann and Hyvärinen, 2010]), as used in various NLP applications [Ma and Collins, 2018], and applied to infomax objectives by van den Oord et al. [2018]. This class of bounds”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “This loss is called InfoNCE [49] as it is inspired by Noise-Contrastive Estimation [24, 45] and has been shown to maximize the mutual information between c i,j and z i + k,j [49].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “In this work, we present MIL-NCE, a new approach marrying the noise contrastive estimation (NCE) framework [29] with multiple instance learning [20].”
Contrastive Multiview Coding (2019, methodology): “struction-based loss functions such as Euclidean losses in different norms e.g. [27], adversarial loss functions [20] that learn the loss in addition to the representation, or contrastive losses e.g. [21, 24, 46, 2] that take advantage of the co-occurence of multiple views. Another recently introduc”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “Although we cannot evaluate p ( x ) or p ( x | c ) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values.”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “[73] propose to use a noise contrastive estimator (NCE) [32] to compare instances instead of classifying them.”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “07 and use NCE with m = 4 , 096 to balance performance and computing cost.”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010) rephrases the density estimation problem as a classification problem, attempting to distinguish data from a known noise distribution.”