The Neural Autoregressive Distribution Estimator

By Hugo Larochelle, Iain Murray. 2011

594 citations recorded by the source corpus.

Subfields: Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Layer Normalization (2016, methodology): “In this experiment, we used the ﬁxed binarization from Larochelle and Murray [2011].”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “The results for our models, along with ones for the Replicated Softmax and DocNADE models from (Salakhutdinov & Hinton, 2009b) and (Larochelle & Lauly, 2012) respectively, are shown in Table 3.”
Deep AutoRegressive Networks (2013, methodology): “As more stochastic or deterministic hidden layers are added to DARN, the advantage of DARN becomes greater as each part of the decoder need only be computed once for DARN per datum, wheras deeper NADE-like models (Uria et al., 2013) require re-computation of large parts of the model for each unit.”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, background): “Larochelle & Murray (2011) showed the effectiveness of learning an optimizer in the few-shot learning setting.”
Music Transformer: Generating Music with Long-Term Structure (2018, background): “…for more details) can be decomposed into a sequence of multi-hot pitch vectors, and their joint probability distributions can be captured using Restricted Boltzmann Machines (Smolensky, 1986; Hinton et al., 2006) or Neural Autoregressive Distribution Estimators (NADE; Larochelle & Murray, 2011).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Since the publication of the ﬁrst formulation of NADE (Larochelle and Murray, 2011), it has been extended to many more settings, other than those described in this paper.”
Image Transformer (2018, background): “The likelihood is made tractable by modeling the joint distribution of the pixels in the image as the product of conditional distributions (Larochelle & Murray, 2011; Theis & Bethge, 2015).”
Unsupervised Visual Representation Learning by Context Prediction (2015, background): “In principle, similar reasoning could be applied in the image domain, a kind of visual “fill in the blank” task, but, again, one runs into the problem of determining whether the predictions themselves are correct [12], unless one cares about predicting only very low-level features [14, 30, 50].”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “WaveNet is one of a family of autoregressive deep generative models that have been applied with great success to data as diverse as text [18], images [17, 26, 19, 28], video [13], handwriting [8] as well as human speech and music.”
Learning to generate chairs with convolutional neural networks (2014, methodology): “This includes a wide variety of methods ranging from Gaussian mixture models [9, 31] to autoregressive models [19] and stochastic variations of neural networks [2, 11, 25, 17, 30].”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “The used datasets are Binary MNIST (Larochelle & Murray, 2011), SVHN (Net-zer et al., 2011), CIFAR10 (Krizhevsky et al., 2009), and ImageNet32 (van den Oord et al., 2016b).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “We distinguish two main types of generative models: likelihood based models, which include VAEs [16, 31], flow based [9, 30, 10, 17] and autoregressive models [20, 39]; and implicit generative ∗Equal contributions.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “The three main classes of powerful explicit density models are autoregressive models [26, 57], flowbased models [8, 9, 21, 16], and probabilistic latent variable models [24, 40, 33].”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “These exact likelihood models are typically speciﬁed as invertible transformations that are parameterized by neural networks (Deco & Brauer, 1995; Larochelle & Murray, 2011; Uria et al., 2013; Dinh et al., 2014; Germain et al., 2015; van den Oord et al., 2016b; Salimans et al., 2017; Chen et al.,…”
Learning to Generate Samples from Noise through Infusion Training (2017, background): “These approaches can be roughly classiﬁed under the following broad categories: • Ordered visible dimension sampling (van den Oord et al., 2016; Larochelle & Murray, 2011).”
Conditional Image Generation with PixelCNN Decoders (2016, methodology): “A similar setup has been used by other autoregressive models such as NADE [14] and RIDE [26].”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Examples of such functions are neural density estimators such as RNNs, NADE Larochelle and Murray (2011); Uria et al. (2013) and MADE (Germain et al., 2015) models.”