RNADE: The real-valued neural autoregressive density-estimator

By Benigno Uria, Iain Murray, Hugo Larochelle. 2013

244 citations recorded by the source corpus.

We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of one-dimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradient-based optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, methodology): “A NADE model for real-valued data could be obtained by applying the derivations shown in Section 2.1 to the Gaussian-RBM (Welling et al., 2005).”
Pixel Recurrent Neural Networks (2016, methodology): “In the literature it is currently best practice to add realvalued noise to the pixel values to dequantize the data when using density functions (Uria et al., 2013).”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et…”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “…exact likelihood models are typically speciﬁed as invertible transformations that are parameterized by neural networks (Deco & Brauer, 1995; Larochelle & Murray, 2011; Uria et al., 2013; Dinh et al., 2014; Germain et al., 2015; van den Oord et al., 2016b; Salimans et al., 2017; Chen et al., 2017).”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “There is a large body of work on neural autoregressive models (Larochelle & Murray, 2011; Uria et al., 2013; Germain et al., 2015; Theis & Bethge, 2015; Oord et al., 2016b; van den Oord et al., 2016d).”
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “Instead of directly modeling the discrete pixel values in an image, it is also possible to de-quantize them by adding noise from the standard uniform distribution, as used by Uria et al. (2013) and others, and modeling the data as being continuous.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Examples of such functions are neural density estimators such as RNNs, NADE Larochelle and Murray (2011); Uria et al. (2013) and MADE (Germain et al., 2015) models.”
Density estimation using Real NVP (2016, methodology): “For examples, the pixel values of an image typically lie in [0 , 256] D after application of the recommended jittering procedure [57, 55].”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “…kernel to match its equilibrium distribution to the data distribution, neural autoregressive distribution estimators [24] (and their recurrent [42] and deep [43] extensions) which decompose the joint distribution into a sequence of tractable conditional distributions over each di-mension,…”
NICE: Non-linear Independent Components Estimation (2014, methodology): “As prescribed in (Uria et al., 2013), we use a dequantized version of the data: we add a uniform noise of 1 256 to the data and rescale it to be in [0, 1] D after dequantization.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “Many types of generative models have ﬂourished in recent years, including likelihood-based generative models, which include autoregressive models (Uria et al., 2013), variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), and invertible ﬂows (Dinh et al., 2014; 2016).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “We dequantized the data following Uria et al. (2013), by adding a uniform noise of 1256 to the data and rescaling it to be in [0, 1] D after dequantization.”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “NADE [19] and RNADE [34] perform a factorization of the output using a predefined and somewhat arbitrary ordering of output dimensions.”
A note on the evaluation of generative models (2015, methodology): “To avoid this case, it is becoming best practice to add real-valued noise to the integer pixel values to dequantize the data (e.g., Uria et al., 2013; van den Oord & Schrauwen, 2014; Theis & Bethge, 2015).”
Generative Image Modeling Using Spatial LSTMs (2015, background): “Several parametrizations have been suggested for the conditional distributions in the context of natural images [5, 15, 41, 44, 45].”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “Deep neural autoregressive models have been applied to image generation for many years, showing promise as a tractable yet expressive density model (Larochelle & Mur-ray, 2011; Uria et al., 2013).”
Auxiliary Deep Generative Models (2016, positive): “To avoid the phenomenon on modeling discretized values with a real-valued estimation (Uria et al., 2013), we added uniform noise between 0 and 1 to each pixel value.”
A Deep and Tractable Density Estimator (2013, methodology): “We dropped the other two datasets tested by Uria et al. (2013), because some of their dimensions only take a ﬁnite number of values even if those are real-valued.”
A Neural Autoregressive Approach to Collaborative Filtering (2016, background): “NADE together with its variants achieved competitive results on many machine learning tasks (Larochelle & Lauly, 2012; Uria et al., 2013; Zheng et al., 2014b; Uria et al., 2014; Zheng et al., 2014a; 2015).”