Autoregressive Energy Machines

By Charlie Nash, Conor Durkan. 2019

58 citations recorded by the source corpus.

Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.

Subfields: Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

Citations in the corpus

Neural Spline Flows (2019, methodology): “For autoregressive transformations, the layers must be masked so as to preserve autoregressive structure, and so we use the ResMADE architecture outlined by Nash and Durkan [39].”
Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (2021, methodology): “Constructing π(τ) ≈ R(τ)/Z, e.g. as per Buesing et al. (2019), MaxEnt RL (Haarnoja et al., 2017), or via an autoregressive method (Nash and Durkan, 2019; Shi et al., 2021) has a particular problem as shown below: if multiple action sequences ~a (i.e. multiple trajectories τ ) lead to a final state…”
GFlowNet Foundations (2021, unknown): “MCMC-based methods have already found some amount of success with learned deep neural networks used to drive sampling (Grathwohl et al., 2021; Dai et al., 2020; Xie et al., 2021; Nash and Durkan, 2019; Seff et al., 2019).”
Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, methodology): “The densities were generated from examples in Nash and Durkan [31].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “This restriction can be removed at the expense of less efficient sampling by learning an autoregressive energy model, for instance, by approximating normalising constants [158] or through score matching [153].”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Formally, we deﬁne hierarchical NCPs by p NCP where each factor is an EBM. p NCP ( z ) resembles energy machines with an autoregressive structure among groups (Nash & Durkan, 2019).”