Neural Autoregressive Distribution Estimation

By Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, Hugo Larochelle. 2016

357 citations recorded by the source corpus.

We present Neural Autoregressive Distribution Estimation (NADE) models, which are neural network architectures applied to the problem of unsupervised distribution and density estimation. They leverage the probability product rule and a weight sharing scheme inspired from restricted Boltzmann machines, to yield an estimator that is both tractable and has good generalization performance. We discuss how they achieve competitive performance in modeling both binary and real-valued observations. We also present how deep NADE models can be trained to be agnostic to the ordering of input dimensions used by the autoregressive product rule decomposition. Finally, we also show how to exploit the topological structure of pixels in images using a deep convolutional architecture for NADE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Related Work The idea of permutation-based AR modeling has been explored in [32, 12], but there are several key differences.”
Music Transformer: Generating Music with Long-Term Structure (2018, background): “Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (e.g., Dong et al., 2018) or as orderless NADEs (Uria et al., 2014; 2016) (e.g., Huang et al., 2017).”
Axial Attention in Multidimensional Transformers (2019, background): “Neural network models in this family have achieved state-of-the-art log likelihoods on highdimensional image and video datasets (van den Oord et al., 2016a; Chen et al., 2018; Menick & Kalchbrenner, 2018; Parmar et al., 2018; Child et al., 2019; Weissenborn et al., 2019; Salimans et al., 2017; Kalch”
Jukebox: A Generative Model for Music (2020, methodology): “Autoregressive generative models such as NADE (Uria et al., 2016), PixelCNN (Van den Oord et al., 2016), and Transformers (Vaswani et al., 2017) use the chain rule of probability to factorize the joint distribution of data into a product of simpler distributions, and ﬂow-based models (Dinh et al.,…”
Conditional Image Generation with PixelCNN Decoders (2016, background): “Recent advances in image modelling with neural networks [30, 26, 20, 10, 9, 28, 6] have made it feasible to generate diverse natural images that capture the high-level structure of the training data.”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “While this makes them powerful models for density estimation [67, 21], their samples often lack global consistency.”
Masked Autoregressive Flow for Density Estimation (2017, background): “There are mainly two families of neural density estimators that are both flexible and tractable: autoregressive models [35] and normalizing flows [24].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Alongside normalizing flows, another popular class of models for high-dimensional distributions is the class of autoregressive models (Bengio and Bengio, 2000; Uria et al., 2016).”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “ussian Mixture models [57,1,16,40,90] and probabilistic graphical models [25,29,87]; and (b) non-parametric approaches such as histograms [65], Splines [72,9], neural network-based density estimation [38,31,79,80,78,54] and Kernel Density Estimation [66,67,37,77]. For an extended review on density e”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, neutral): “The parameters can be learned via maximum likelihood estimation (MLE) as in the case of autoregressive models [16], normalizing flows [17], and variational autoencoders [7, 18], or via adversarial training e.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “Autoregressive models (e.g. Schmidhuber & Heil 1994; Rosenfeld 2000; Bengio et al. 2003; Graves 2013; van den Oord et al. 2016b; Uria et al. 2016) estimate the density of an example X ∈ R by decomposing it sequentially using”
Analyzing Inverse Problems with Invertible Neural Networks (2018, methodology): “MNIST, CelebA, LSUN bedrooms), and some encouraging results for conditional modeling exist as well [29, 38, 31, 45].”
Counterpoint by Convolution (2019, methodology): “To model the joint probability distribution of the multi-hot pitch vector x t , they employ a Restricted Boltzmann Machine (RBM [19, 32]) or Neural Autoregressive Distribution Estimator (NADE [25]) at each time step.”
Mining gold from implicit models to improve likelihood-free inference (2018, background): “6), summing over x ∈ [5, 15], versus the training sample size (which refers to the total number of x samples, distributed over 10 values of θ ∈ [−1,−0.”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, methodology): “Neural Autoregressive Density Estimator (NADE) [34] is a prominent earlier work followed by more recent Wavenet [35], SampleRNN [36] and MelNet [37].”
Fast likelihood-free cosmology with neural density estimators and active learning (2019, background): “(2) Neural autoregressive density estimators construct parametric densities for this set of one-dimensional conditionals, where the parameters of each of the conditionals are parameterized as a neural network (Uria et al. 2016).”
Deep Learning Based Recommender System (2017, background): “• Neural Autoregressive Distribution Estimation (NADE) [81, 152] is an unsupervised neural network built atop autoregressive model and feedforward neural networks.”
Machine learning for digital soil mapping: Applications, challenges and suggested solutions (2020, background): “This can take the form of an auto-regressive model between the predictions (Uria et al., 2016).”