On the Equivalence between Deep NADE and Generative Stochastic Networks

By L. Yao, Sherjil Ozair, Kyunghyun Cho, Yoshua Bengio. 2014

9 citations recorded by the source corpus.

Neural Autoregressive Distribution Estimators (NADEs) have recently been shown as successful alternatives for modeling high dimensional multimodal distributions. One issue associated with NADEs is that they rely on a particular order of factorization for P(x). This issue has been recently addressed by a variant of NADE called Orderless NADEs and its deeper version, Deep Orderless NADE. Orderless NADEs are trained based on a criterion that stochastically maximizes P(x) with all possible orders of factorizations. Unfortunately, ancestral sampling from deep NADE is very expensive, corresponding to running through a neural net separately predicting each of the visible variables given some others. This work makes a connection between this criterion and the training criterion for Generative Stochastic Networks (GSNs). It shows that training NADEs in this way also trains a GSN, which defines a Markov chain associated with the NADE model. Based on this connection, we show an alternative way to sample from a trained Orderless NADE that allows to trade-off computing time and quality of the samples: a 3 to 10-fold speedup (taking into account the waste due to correlations between consecutive samples of the chain) can be obtained without noticeably reducing the quality of the samples. This is achieved using a novel sampling procedure for GSNs called annealed GSN sampling, similar to tempering methods that combines fast mixing (obtained thanks to steps at high noise levels) with accurate samples (obtained thanks to steps at low noise levels).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GSNs : Generative Stochastic Networks (2015, methodology): “In that spirit, the extreme solution is to completely eliminate latent variables, with models that can, however, still perform very well, like NADE (Larochelle & Murray, 2011) or even recurrent neural networks (if the stationarity assumption makes sense).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “ provide state of the art unsupervised learning results on an extremely complex dataset, natural images. Other related methods for training probabilistic models include generative stochastic networks [3,46] which directly train a Markov kernel to match its equilibrium distribution to the data distri”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, background): “2014)) and in testing (e.g. by handling the components in chunks rather than fully sequentially). The computational efﬁciency of sampling for NADE-kcan be further improved based on the recent work of Yao et al. (2014) where an annealed Markov chain may be used to efﬁciently generate samples from the”
Counterpoint by Convolution (2019, methodology): “Moreover, we show that even the cheap approximate blocked Gibbs procedure from [40] yields better samples than ancestral sampling, based on both log-likelihood and human evaluation.”