A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

By Tim Salimans. 2016

20 citations recorded by the source corpus.

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability necessary for efficient stochastic gradient variational inference. To learn the parameters of the new model, we approximate the posterior of the latent variables with a variational auto-encoder. Rather than making the usual mean-field assumption however, the encoder parameterizes a new type of structured variational approximation that retains the prior dependencies of the generative model. Using this structured posterior approximation, we are able to perform joint training of deep models with many layers of latent random variables, without having to resort to stacking or other layerwise training procedures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “We compare a number of inference procedures; each either bottom-up, or top-down (see also Salimans (2016); Kaae Sønderby et al. (2016)).”
Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “s is not a new concept and has been previously done with Gaussian distributions in the context of generative modelling (Hinton & Ghahramani, 1997; Harva & Kabán, 2007; Salimans, 2016) and with logistic distributions at (Bengio et al.”
Discrete Variational Autoencoders (2016, methodology): “We use the same autoregressive variable order for the approximating posterior as for the Published as a conference paper at ICLR 2017 prior, as in DRAW (Gregor et al., 2015), variational recurrent neural networks (Chung et al., 2015), the deep VAE of Salimans (2016), and ladder networks (Rasmus et a”
An Introduction to Variational Autoencoders (2019, background): “This idea is explored by (Salimans, 2016) and (Sønderby et al.”