MADE: Masked Autoencoder for Distribution Estimation

By Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle. 2015

978 citations recorded by the source corpus.

There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “Related Work The idea of permutation-based AR modeling has been explored in [32, 12], but there are several key differences.”
Neural Autoregressive Distribution Estimation (2016, methodology): “Gregor et al. (2015) later exploited the VAE approach to develop DRAW, a directed generative model for images based on a read-write attentional mechanism.”
Pixel Recurrent Neural Networks (2016, methodology): “Similar masks have also been used in (variational) autoen-coders (Gregor et al., 2014; Germain et al., 2015).”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “WaveNet [27] is a convolutional autoregressive model which produces all p ( x t | x t ) in one forward pass, by making use of causal —or masked —convolutions [19, 6].”
Predictive Sampling with Forecasting Autoregressive Models (2020, background): “Numerous follow up works by Germain et al. (2015); van den Oord et al.”
Stochastic Normalizing Flows (2020, background): “…families: (i) Coupling layers (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; M ¨ uller et al., 2018), which are a subclass of autoregressive ﬂows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual ﬂows (Chen et…”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “…exact likelihood models are typically speciﬁed as invertible transformations that are parameterized by neural networks (Deco & Brauer, 1995; Larochelle & Murray, 2011; Uria et al., 2013; Dinh et al., 2014; Germain et al., 2015; van den Oord et al., 2016b; Salimans et al., 2017; Chen et al., 2017).”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “Autoregressive models (Germain et al., 2015; Oord et al., 2016) directly specify the joint distribution p ( x ) as a sequence of explicit conditional distributions using the product rule.”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “This class of models includes PixelRNN and PixelCNN (van den Oord et al., 2016c;b), MAF (Papa-makarios et al., 2017), MADE (Germain et al., 2015), and many others.”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “There is a large body of work on neural autoregressive models (Larochelle & Murray, 2011; Uria et al., 2013; Germain et al., 2015; Theis & Bethge, 2015; Oord et al., 2016b; van den Oord et al., 2016d).”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “For CIFAR-10, we use convolutional MADE van den Oord et al. (2016) with zero, one or two hidden layers.”
Density estimation using Real NVP (2016, background): “The sequential nature of this model limits its computational efficiency.”
GAN Inversion: A Survey (2021, methodology): “DARs [160], [161] are feed-forward models which give predictive probability distribution P (xt+1|x1, x2, · · · , xt) for future values xt+1 from past observations x1, x2, · · · , xt.”
Gradient-Based Neural DAG Learning (2019, methodology): “GraN-DAG has strong connections with MADE (Germain et al., 2015), a method used to learn distributions using a masked NN which enforces the so-called autoregressive property .”
Variational Lossy Autoencoder (2016, positive): “) ∈ R + are general functions that can be parametrized by expressive neural networks, such as MADE and PixelCNN variants (Germain et al., 2015; van den Oord et al., 2016a).”
Neural Photo Editing with Introspective Adversarial Networks (2016, methodology): “We take advantage of the auto-encoding nature of our architecture and implement the MADE [14] variant of Inverse Autoregressive Flow (IAF) [15].”
Professor Forcing: A New Algorithm for Training Recurrent Networks (2016, methodology): “…NLL DBN 2hl (Germain et al. , 2015) ≈ 84.55 NADE (Larochelle and Murray, 2011) 88.33 EoNADE-5 2hl (Raiko et al. , 2014) 84.68 DLGM 8 leapfrog steps (Salimans et al. , 2014) ≈ 85.51 DARN 1hl (Gregor et al. , 2015) ≈ 84.13 DRAW (Gregor et al. , 2015) ≤ 80.97 Pixel RNN (van den Oord et al. , 2016) 79.”
PixelVAE: A Latent Variable Model for Natural Images (2016, neutral): “(Germain et al., 2015).”