Multimodal Transitions for Generative Stochastic Networks

By Sherjil Ozair, L. Yao, Yoshua Bengio. 2013

11 citations recorded by the source corpus.

Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distribution. The result of training is therefore a machine that generates samples through this Markov chain. However, the previously introduced GSN consistency theorems suggest that in order to capture a wide class of distributions, the transition operator in general should be multimodal, something that has not been done before this paper. We introduce for the first time multimodal transition distributions for GSNs, in particular using models in the NADE family (Neural Autoregressive Density Estimator) as output distributions of the transition operator. A NADE model is related to an RBM (and can thus model multimodal distributions) but its likelihood (and likelihood gradient) can be computed easily. The parameters of the NADE are obtained as a learned function of the previous state of the learned Markov chain. Experiments clearly illustrate the advantage of such multimodal transition distributions over unimodal GSNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GSNs : Generative Stochastic Networks (2015, methodology): “One possible solution to this problem is to use a multimodal reconstruction distribution such as in Ozair et al. (2014), Larochelle & Murray (2011) or Dinh et al. (2015).”
Conditional generative adversarial nets for convolutional face generation (2015, methodology): “This work has been further developed under the label of generative stochastic networks [3, 18], where the process of noising C(X̃ | X) is generalized to one of decoding into a hidden state using observed data.”
General Stochastic Networks for Classification (2014, unknown): “When applied to image reconstruction, they were able to outperform various baseline systems, due to its ability to learn multi-modal representations [17, 21].”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, background): “The only way to avoid that would be to make P ( X h ) a highly multi-modal complicated distribution (and this is the route that Ozair et al. | (2014) have taken).”