Conditional Noise-Contrastive Estimation of Unnormalised Models

By Ciwan Ceylan, Michael U Gutmann. 2018

50 citations recorded by the source corpus.

Many parametric statistical models are not properly normalised and only specified up to an intractable partition function, which renders parameter estimation difficult. Examples of unnormalised models are Gibbs distributions, Markov random fields, and neural network models in unsupervised deep learning. In previous work, the estimation principle called noise-contrastive estimation (NCE) was introduced where unnormalised models are estimated by learning to distinguish between data and auxiliary noise. An open question is how to best choose the auxiliary noise distribution. We here propose a new method that addresses this issue. The proposed method shares with NCE the idea of formulating density estimation as a supervised learning problem but in contrast to NCE, the proposed method leverages the observed data when generating noise samples. The noise can thus be generated in a semi-automated manner. We first present the underlying theory of the new method, show that score matching emerges as a limiting case, validate the method on continuous and discrete valued synthetic data, and show that we can expect an improved performance compared to NCE when the data lie in a lower-dimensional manifold. Then we demonstrate its applicability in unsupervised deep learning by estimating a four-layer neural image model.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “While this method has been recently extended (Ceylan & Gutmann, 2018), these methods are challenging to extend to high-dimensional data.”
Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling (2020, methodology): “The linear Independent Components Analysis (ICA) model is commonly used to quantitatively evaluate the performance of methods for training unnormalized models (Gutmann & Hyv¨arinen, 2010; Hyv¨arinen, 2005; Ceylan & Gutmann, 2018).”
How to Train Your Energy-Based Models (2021, methodology): “As one unique feature that Contrastive Divergence and Score Matching do not have, NCE provides the normalizing constant of an Energy-Based Model as a by-product of its training procedure.”
Autoregressive Energy Machines (2019, background): “These include contrastive divergence (Hinton, 2002), noise-contrastive estimation (Gut-mann & Hyv ¨ arinen, 2010; Ceylan & Gutmann, 2018), and score matching (Hyv¨arinen, 2005).”
Exponential Family Estimation via Adversarial Dynamics Embedding (2019, background): “Noise-contrastive estimation (Gutmann and Hyvärinen, 2010; Ceylan and Gutmann, 2018) recasts the problem as ratio estimation between the target distribution and a pre-defined auxiliary noise distribution.”
A Cookbook of Self-Supervised Learning (2023, methodology): “Specifically, they propose using the InfoNCE loss [Gutmann and Hyvärinen, 2010, Ceylan and Gutmann, 2018] to compare the representations of the clean and corrupted inputs.”