Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

By Michael U Gutmann, Aapo Hyvärinen. 2012

893 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Dual Contradistinctive Generative Autoencoder (2020, neutral): “This can be achieved by learning to contrast (classify) the set of training images with the set of generated images [60, 19, 16].”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “All of these methods rely on noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2012), which underlies the state-ofthe-art in self-supervised representation learning (Oord et al., 2018; Chen et al., 2020), in which identifiability has been used as an analysis tool (Roeder et al., 2021; Zimm”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, background): “GANs are sample-based, that is, samples are drawn from the model for learning [23, 18].”
Pointing the Unknown Words (2016, neutral): “The hierarchical softmax (Morin and Bengio, 2005), importance sampling (Bengio and Sen´ecal, 2008; Jean et al., 2014), and the noise contrastive estimation (Gutmann and Hyv¨arinen, 2012; Mnih and Kavukcuoglu, 2013) methods are in the class.”
A fast and simple algorithm for training neural probabilistic language models (2012, methodology): “Following Gutmann and Hyv¨arinen (2012), we assume that noise samples are k times more frequent than data samples, so that datapoints come from the mixture Then the posterior probability that sample w came from the data distribution is Since we would like to ﬁt P hθ to P hd , we use P hθ in place…”
Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks (2015, methodology): “To address this problem, [30] proposed a simple yet powerful method called Negative sampling which is a simplified version of Noise Contrastive Extimation (NCE)[31].”
Deep Learning: Methods and Applications (2014, methodology): “The reported algorithm makes use of a method called noise-contrastive estimation or NCE [139] to achieve much faster training for NNLMs, with time complexity independent of the vocabulary size; hence a flat instead of tree-structured output layer in the NNLM is used.”
Relevance-based Word Embedding (2017, methodology): “Similar to [24], we propose to use the noise contrastive estimation (NCE) [12] which hypothesizes that we can achieve a good model by only dierentiating the data from noise via a logistic regression model.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “[2013], Gutmann and Hyvärinen [2012]. Concretely, we draw a sample s ∼ ρe , create a random perturbation of s to get a point s̃ 6∈ U .”
Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, methodology): “We further provide a practical algorithm for estimating the model using the idea of contrastive learning [10, 15, 14], and prove its consistency.”
Learning in Implicit Generative Models (2016, methodology): “This piques our interest in more general approaches for non-maximum likelihood and likelihood-free estimation methods, of which there is much work (Frogner et al., 2015; Gutmann and Hyvärinen, 2012; Hall, 2005; Lyu, 2011; Marin et al., 2012).”
Matching the Blanks: Distributional Similarity for Relation Learning (2019, methodology): “pare every pair of relation statements, as in Equation 1, and so we use a noise-contrastive estimation (Gutmann and Hyvärinen, 2012; Mnih and Kavukcuoglu, 2013).”
Revisiting Classifier Two-Sample Tests (2016, neutral): “).”
Variational Approaches for Auto-Encoding Generative Adversarial Networks (2017, background): “This is a principle of learning known under various names, including adversarial training [11], estimation-by-comparison [13, 14], and unsupervised-as-supervised learning [15].”
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (2018, methodology): “There are clear connections to noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2012), a negative sampling method for parameter estimation in joint models of the form p(y) = exp (s(y; ✓)) Z(✓) ; Z(✓) = X”