An Architecture for Deep, Hierarchical Generative Models

By Philip Bachman. 2016

54 citations recorded by the source corpus.

We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Model Architectures

Citations in the corpus

The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Beyond raw audio generation, several generative models that capture large-scale structure have been proposed for images [2, 3, 12, 27], dialogue [40], 3D shapes [30] and symbolic music [37].”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Finally, Bachman [57] and Kingma et al. [56] explore hierarchical factorizations/architectures mainly to improve generative modeling performance (in terms of testing log -likelihood), rather than exploring it from a representation learning perspective.”
Taming VAEs (2018, background): “Deep variational auto-encoders (VAEs) have made substantial progress in the last few years on modelling complex data such as images [1, 2, 3, 4, 5, 6, 7], speech synthesis [8, 9], molecular discovery [10, 11, 12], robotics [13, 14, 15] and 3d-scene understanding [16, 17, 18].”
Towards Deeper Understanding of Variational Autoencoding Models (2017, background): “(Kingma & Welling, 2013; Jimenez Rezende et al., 2014; Kingma et al., 2016; Kaae Sønderby et al., 2016; Bachman, 2016; Gulrajani et al., 2016).”
A Hybrid Convolutional Variational Autoencoder for Text Generation (2017, background): “While training VAE-based models seems to pose little difﬁculty when applied to the tasks of generating natural images (Bachman, 2016; Gul-rajani et al., 2016) and speech (Fraccaro et al., 2016), their application to natural text generation requires additional care (Bowman et al., 2016; Miao et al.,…”
Z-Forcing: Training Stochastic Recurrent Networks (2017, background): “…with respect to strong baselines, e.g. DRAW (Gregor et al., 2015), and is outperformed by deeper version of autoregressive models with latent variables, i.e. PixelVAE (gated) (Gulrajani et al., 2016), and deep autoregressive models such as PixelRNN (Oord et al., 2016) and MatNets (Bachman, 2016).”
Online Learning with Gated Linear Networks (2017, result): “To the best of our knowledge, this result matches the state of the art (Van Den Oord et al., 2016) of any batch trained density model which outputs exact probabilities, and is close to the estimated upper bounds reported for state of the art variational approaches (Bachman, 2016).”
Twin Networks: Matching the Future for Sequence Generation (2017, unknown): “015) 80:97 P-Forcing (3-layer) (Lamb et al., 2016) 79:58 PixelRNN (1-layer) (Oord et al., 2016b) 80:75 PixelRNN (7-layer) (Oord et al., 2016b) 79:20 PixelVAE (Gulrajani et al., 2016) 79:02H MatNets (Bachman, 2016) 78:50H Baseline LSTM (3-layers) 79:87 + TwinNet (3-layers) 79:35 Baseline LSTM (3-laye”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, background): “…by guest on 04 June 2021 (2014), Kingma and Welling (2014), Bengio, Thibodeau-Laufer, Alain, and Yosinski (2014), Kulkarni, Whitney, Kohli, and Tenenbaum (2015) and, more recently, Bachman (2016), all of which use promising generative-based modeling techniques.”