Iterative Neural Autoregressive Distribution Estimator NADE-k

By Tapani Raiko, L. Yao, Kyunghyun Cho, Yoshua Bengio. 2014

41 citations recorded by the source corpus.

Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “A generalization of NADE based on this connection to mean ﬁeld inference has been further explored by Raiko et al. (2014).”
Pixel Recurrent Neural Networks (2016, neutral): “, 2014), [4] (Raiko et al., 2014), [5] (Rezende et al.”
DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “The network is similar to the recently introduced Recurrent Attention Model (RAM) (Mnih et al., 2014), except that our attention method is differentiable; we therefore refer to it as “Differ-entiable RAM”.”
Ladder Variational Autoencoders (2016, methodology): “Other directions for future work include extending these models to semi-supervised learning which will likely beneﬁt form the learned deep structured hierarchies of latent variables and studying more elaborate inference schemes such as a k -step iterative inference in the LVAE [14].”
The Variational Gaussian Process (2015, neutral): “, 2015), [3] (Rezende & Mohamed, 2015), [4] (Raiko et al., 2014), [5] (Murray & Salakhutdinov, 2009), [6] (Gregor et al.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, neutral): “A similar behavior was observed earlier with another generative model using iterative refinement (Raiko et al., 2014), which we leave as a future work to investigate.”
How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks (2016, background): “A further extension could be to make the inference in k steps over an iterative inference procedure (Raiko et al., 2014).”
Recurrent Ladder Networks (2017, methodology): “Iterative inference with neural architectures was previously proposed in deep Boltzmann machines [10], NADE-k [23] and recently for VAE [14].”
Z-Forcing: Training Stochastic Recurrent Networks (2017, unknown): “…f ( a ) have a single hidden layer, with 1024 units for Blizzard and 512 (Germain et al., 2015) ≈ 84.55 NADE (Uria et al., 2016) 88.33 EoNADE-5 2hl (Raiko et al., 2014) 84.68 DLGM 8 (Salimans et al., 2014) ≈ 85.51 DARN 1hl (Gregor et al., 2015) ≈ 84.13 DRAW (Gregor et al., 2015) ≤ 80.97 PixelVAE…”