Learning Hierarchical Features from Generative Models

By Shengjia Zhao, Jiaming Song, Stefano Ermon. 2017

78 citations recorded by the source corpus.

Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning tasks. Generative models, on the other hand, have benefited less from hierarchical models with multiple layers of latent variables. In this paper, we prove that hierarchical latent variable models do not take advantage of the hierarchical structure when trained with existing variational methods, and provide some limitations on the kind of features existing models can learn. Finally we propose an alternative architecture that do not suffer from these limitations. Our model is able to learn highly interpretable and disentangled hierarchical features on several natural image datasets with no task specific regularization or prior knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefano Ermon
2. Jiaming Song
3. Sergey Levine
4. Pieter Abbeel
5. Jonathan Ho
6. Chelsea Finn
7. Ruslan Salakhutdinov
8. Yang Song

In Computer Vision and Pattern Recognition

1. Stefano Ermon
2. Luc Van Gool
3. Jiaming Song
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Trevor Darrell
7. Pieter Abbeel
8. Sergey Levine

In Generative Models

1. Stefano Ermon
2. Jiaming Song
3. Jonathan Ho
4. Arash Vahdat
5. Rewon Child
6. Karsten Kreis
7. Pieter Abbeel
8. Yang Song

In Representation & Self-Supervised Learning

1. Stefano Ermon
2. Jiaming Song
3. Rewon Child
4. Shengjia Zhao
5. Greg Ver Steeg
6. Aram Galstyan
7. Shuyang Gao
8. Rob Brekelmans

Citations in the corpus

• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “A.5 A NOTE ON LEARNING HIERARCHICAL FEATURES The work of Zhao et al. (2017) may appear to contradict our work, by suggesting that additional layers in hierarchical VAEs do not lead to additional expressivity, based off their finding that Gibbs sampling from the last stochastic layer is sufficient…”
• InfoVAE: Information Maximizing Variational Autoencoders (2017, methodology): “However, autoregressive density estimators such as PixelRNN/PixelCNN do not explicitly include latent variables designed to capture meaningful latent features, which is often the main purpose of unsupervised learning [35].”
• Degeneration in VAE: in the Light of Fisher Information Loss (2018, background): “In previous work, such as Zhao et al. (2017), Mutual Information is measured to reinforce the connection between x and z.”
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables (2019, background): “Especially when using deep hierarchies of latent variables, the higher latent layers can become redundant and therefore unused (Zhao et al., 2017).”
• An Interpretable Generative Model for Handwritten Digits Synthesis (2019, background): “It is desired to provide explanable models for both VAEs and GANs.”
• Auto-Encoding Total Correlation Explanation (2018, methodology): “(Sønderby et al., 2016; Zhao et al., 2017b) highlight limitations of the naive hierarchical VAE, such as representational inefficiency, and propose alternative ladder neural network structures for learning hierarchical features.”