A Deep and Tractable Density Estimator

By Benigno Uria, Iain Murray, Hugo Larochelle. 2013

202 citations recorded by the source corpus.

The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep AutoRegressive Networks (2013, methodology): “As more stochastic or deterministic hidden layers are added to DARN, the advantage of DARN becomes greater as each part of the decoder need only be computed once for DARN per datum, wheras deeper NADE-like models (Uria et al., 2013) require re-computation of large parts of the model for each unit.”
Music Transformer: Generating Music with Long-Term Structure (2018, background): “Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (e.g., Dong et al., 2018) or as orderless NADEs (Uria et al., 2014; 2016) (e.g., Huang et al., 2017).”
Neural Autoregressive Distribution Estimation (2016, background): “Other choices, like single variable-variance Gaussians, sinh-arcsinh distributions, and mixtures of Laplace distributions, have been examined by Uria (2015).”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et…”
Variational Inference with Normalizing Flows (2015, methodology): “We used the binarized dataset as in (Uria et al., 2014).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “…match its equilibrium distribution to the data distribution, neural autoregressive distribution estimators [24] (and their recurrent [42] and deep [43] extensions) which decompose the joint distribution into a sequence of tractable conditional distributions over each di-mension, adversarial…”
DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “After a ﬁxed number of glimpses the network uses a softmax layer to classify the MNIST digit.”
Generative Image Modeling Using Spatial LSTMs (2015, background): “Several parametrizations have been suggested for the conditional distributions in the context of natural images [5, 15, 41, 44, 45].”
Training generative neural networks via Maximum Mean Discrepancy optimization (2015, methodology): “A recent example is the Neural Autoregressive Density Estimator due to Uria, Murray, and Larochelle [15].”
Masked Autoregressive Flow for Density Estimation (2017, background): “An ensemble of 32 Deep RNADEs was reported to achieve 157.0 nats [32].”
Robust deep learning based protein sequence design using ProteinMPNN (2022, methodology): “To enable application to a broad range of single and multi-chain design problems, we replaced the fixed N to C terminal decoding order with an order agnostic autoregressive model in which the decoding order is randomly sampled from the set of all possible permutations (8).”
A Neural Autoregressive Approach to Collaborative Filtering (2016, methodology): “The factors in front of the sum come from the fact that the total number of elements in the sum is D and that we are averaging over D − i +1 possible choices for the item at position i , similar to (Uria et al., 2014) and (Zheng et al., 2015).”
Neural Variational Inference for Text Processing (2015, background): “Another class of neural generative models make use of the autoregressive assumption (Larochelle & Murray, 2011; Uria et al., 2014; Germain et al., 2015; Gregor et al., 2014).”
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models (2014, methodology): “The images are of size 28×28, and we use the binarised data set from Larochelle & Murray (2011).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Examples of RNN-based autoregressive models include distribution estimators (Larochelle and Murray, 2011; Uria et al., 2013, 2014), sequence models (Mikolov et al., 2010; Graves, 2013; Sutskever et al., 2014), and image/video models (Theis and Bethge, 2015; Oord et al., 2016; Kalchbrenner et al.,…”
Reweighted Wake-Sleep (2014, methodology): “1 MNIST We use the MNIST dataset that was binarized according to Murray and Salakhutdinov (2009) and downloaded in it’s binarized form from (Larochelle, 2011).”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, methodology): “D (Uria et al. , 2014).”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “ussian Mixture models [57,1,16,40,90] and probabilistic graphical models [25,29,87]; and (b) non-parametric approaches such as histograms [65], Splines [72,9], neural network-based density estimation [38,31,79,80,78,54] and Kernel Density Estimation [66,67,37,77]. For an extended review on density e”
Deep Directed Generative Autoencoders (2014, methodology): “ proposed model, and analyze the inﬂuence of various techniques on the performance of the model. We used the binarized MNIST handwritten digits dataset. We used the same binarized version of MNIST as Murray and Larochelle (2014), and also used the same training-validation-test split. We trained shal”