Deep AutoRegressive Networks

By Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, Daan Wierstra. 2013

307 citations recorded by the source corpus.

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets, including several UCI data sets, MNIST and Atari 2600 games.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Generative Adversarial Nets (2014, neutral): “In practice, equation”
Auto-Encoding Variational Bayes (2013, neutral): “The recently proposed DARN method [GMW13], also learns a directed probabilistic model using an auto-encoding structure, however their method applies to binary latent variables.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “Interestingly, single-layer fDARN ( Gregor et al., 2013) models, which have autoregressive connections between the latent variables, perform better than any of the SBN models trained using the same al-gorithm.”
Neural Autoregressive Distribution Estimation (2016, methodology): “For these experiments, in addition to the baselines already described in Section 7.1, we consider the following: • DARN (Gregor et al., 2014): This deep generative autoencoder has two hidden layers, one deterministic and one with binary stochastic units.”
Pixel Recurrent Neural Networks (2016, methodology): “Similar masks have also been used in (variational) autoen-coders (Gregor et al., 2014; Germain et al., 2015).”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et al., 2014).”
Neural Discrete Representation Learning (2017, methodology): “Our work also extends the line of research where autoregressive distributions are used in the decoder of VAEs and/or in the prior [14].”
Conditional Image Generation with PixelCNN Decoders (2016, background): “Recent advances in image modelling with neural networks [30, 26, 20, 10, 9, 28, 6] have made it feasible to generate diverse natural images that capture the high-level structure of the training data.”
Towards Conceptual Compression (2016, methodology): “We also experiment with making convolutional DRAW hierarchical in a similar way that we would build conventional deep variational auto-encoders (Gregor et al., 2014) – stacking more layers of latent and deterministic variables.”
Variational Inference with Normalizing Flows (2015, methodology): “…in semi-supervised classification (Kingma et al., 2014), drives the models that currently produce the most realistic generative models of images (Gregor et al., 2014; 2015; Rezende et al., 2014; Kingma & Welling, 2014), and are a default tool for the understanding Proceedings of the 32nd…”
GSNs : Generative Stochastic Networks (2015, background): “More recently, it was followed up by the various variational auto-encoders or VAEs (Gregor et al., 2014; Kingma & Welling, 2014; Mnih & Gregor, 2014; Rezende et al., 2014), and related directed models (Bornschein & Bengio, 2014; Ozair & Bengio, 2014).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “In [21, 11, 33, 31] variational learning and inference algorithms were developed which allow a ﬂexi-ble generative model and posterior distribution over latent variables to be directly trained against each other.”
NICE: Non-linear Independent Components Estimation (2014, background): “However, recent advances in the framework of variational auto-encoders (VAE) - which come under a variety of names and variants (Kingma and Welling, 2014; Rezende et al., 2014; Mnih and Gregor, 2014; Gregor et al., 2014) - allowed effective approximate inference for training.”
Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “in [7]; both amortize inference by using an inference network.”
DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “…[2] (Murray & Salakhutdinov, 2009), [3] (Uria et al., 2014), [4] (Raiko et al., 2014), [5] (Rezende et al., 2014), [6] (Salimans et al., 2014), [7] (Gregor et al., 2014 Once the DRAW network was trained, we generated MNIST digits following the method in Section 2.3, examples of which are presented…”
Categorical Reparameterization with Gumbel-Softmax (2016, background): “• DARN (Gregor et al., 2013) uses b = f(z̄) + f ′(z̄)(z̄ − z), where the baseline corresponds to the first-order Taylor approximation of f(z) from f(z̄).”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, neutral): “As stated above, our system belongs to the family of VAEs and is generally similar to recent work such as Kingma and Welling [7], Rezende et al. [8], Gregor et al. [26], Bayer and Osendorfer [27].”
Importance Weighted Autoencoders (2015, background): “Other models are deﬁned in terms of belief networks (Neal, 1992; Gregor et al., 2014).”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “For models with discrete latent variables, the discontinuous nature of which makes reparameterization not useful, a number of score function estimators have been developed [31, 17, 33, 28, 42, 18], which differ primarily in the variance reduction techniques used.”