Energy-Inspired Models: Learning with Sampler-Induced Distributions

By Dieterich Lawson, George Tucker, Bo Dai, Rajesh Ranganath. 2019

32 citations recorded by the source corpus.

Energy-based models (EBMs) are powerful probabilistic models, but suffer from intractable sampling and density evaluation due to the partition function. As a result, inference in EBMs relies on approximate sampling algorithms, leading to a mismatch between the model and inference. Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model. This yields a class of energy-inspired models (EIMs) that incorporate learned energy functions while still providing exact samples and tractable log-likelihood lower bounds. We describe and evaluate three instantiations of such models based on truncated rejection sampling, self-normalized importance sampling, and Hamiltonian importance sampling. These models outperform or perform comparably to the recently proposed Learned Accept/Reject Sampling algorithm and provide new insights on ranking Noise Contrastive Estimation and Contrastive Predictive Coding. Moreover, EIMs allow us to generalize a recent connection between multi-sample variational lower bounds and auxiliary variable variational inference. We show how recent variational bounds can be unified with EIMs as the variational family.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “Other methods for learning transition operators of Markov chains include infusion training [2], variational walkback [15], generative stochastic networks [1], and others [50, 54, 36, 42, 35, 65].”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Also, prior work on VAEs has shown that learned re-sampling or energy-based transformation of their priors can be effective [55, 56].”
Energy-Based Models for Deep Probabilistic Regression (2020, methodology): “Recently, energy-based models have been used primarily for unsupervised learning problems within computer vision [58, 14, 11, 29, 40], where DNNs are directly used to predict Eθ(x).”
Training Deep Energy-Based Models with f-Divergence Minimization (2020, methodology): “…reparametrization method in Section 3.1.2, we may redeﬁne the generative process as the combination of a proposal distribution and MCMC sampling so that we can jointly train the proposal model and the energy function, and we only need to backpropagate through a few MCMC steps (Lawson et al., 2019).”
KALE: When Energy-Based Learning Meets Adversarial Training (2020, background): “In a different line of work, (Lawson et al., 2019) propose to treat the sampling procedure as a model of interest which is then learned by maximizing the ELBO.”
Generalized Energy Based Models (2020, methodology): “Ding et al. (2019) perform sampling on the feature space of some auxiliary pre-trained network; while Lawson et al. (2019) treat the sampling procedure as a model on its own, learned by maximizing the ELBO.”
Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation (2019, background): “Several recent works have viewed Monte Carlo VI bounds through the lens of augmented VI [3, 10, 20, 11, 16].”