Early Visual Concept Learning with Unsupervised Deep Learning

By Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, Alexander Lerchner. 2016

184 citations recorded by the source corpus.

Automated discovery of early visual concepts from raw image data is a major open challenge in AI research. Addressing this problem, we propose an unsupervised approach for learning disentangled representations of the underlying factors of variation. We draw inspiration from neuroscience, and show how this can be achieved in an unsupervised generative model by applying the same learning pressures as have been suggested to act in the ventral visual stream in the brain. By enforcing redundancy reduction, encouraging statistical independence, and exposure to data with transform continuities analogous to those to which human infants are exposed, we obtain a variational autoencoder (VAE) framework capable of learning disentangled factors. Our approach makes few assumptions and works well across a wide variety of datasets. Furthermore, our solution has useful emergent properties, such as zero-shot inference and an intuitive understanding of "objectness".

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Variational Information Bottleneck (2017, methodology): “(The VAE objective, but with different values of β, was also explored in Higgins et al. (2016), but from a different perspective.) The method of Wang et al. (2016b), which was submitted to ICLR17 concurrently with this paper, proposes a latent variable generative model of both x and y; their variati”
Towards Deep Symbolic Reinforcement Learning (2016, background): “In particular, a more sophisticated deep network capable of unsupervised learning of disentangled representations (having a compositional structure) will be required to handle more realistic images than occur in our benchmark games (eg: [23]).”
Probabilistic Model-Agnostic Meta-Learning (2018, methodology): “, with VAEs), we put a multiplier β in front of the KL-divergence against the prior [17] in Algorithm 1.”
On Variational Bounds of Mutual Information (2019, background): “Many recent papers in representation learning have focused on learning latent representations in a generative model that correspond to human-interpretable or “disentangled” concepts (Higgins et al., 2016; Burgess et al., 2018; Chen et al., 2018; Kumar et al., 2017).”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, methodology): “We study this question using β-VAEs [25, 30] which is a natural extension of VAEs and have been shown to provide good disentanglement properties.”
Disentangled Sequential Autoencoder (2018, methodology): “We also include two baseline results from Hsu et al. (2017): one used the i-vector method (Dehak et al.”
Comparing continual task learning in minds and machines (2018, methodology): “Unlike a standard VAE, which typically learns an unstructured latent representation, the β-VAE can learn disentangled and interpretable data generating factors, similar to a child who acquires structured visual concepts through passive observation of its surroundings (30).”
Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations (2018, background): “The behavior of these models has been widely studied [4, 5, 6, 7] and a clear trade-off between desirable latent space properties (such as disentanglement of generative factors, linearity & sparsity) and reconstruction quality is usually present.”
InfoVAE: Information Maximizing Variational Autoencoders (2017, methodology): “A similar model from the perspective of denoising autoencoders has been proposed in [3, 1] and from the perspective of modeling properties of human visual learning in [11].”
Learning by Abstraction: The Neural State Machine (2019, background): “Several works have explored the discovery and use of visual concepts in the contexts of reinforcement or unsupervised learning [35, 17] as well as in classical computer vision [21, 77].”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, background): “etworks can be quite correlated which might impair the estimate of the variance in the normalizer. More independent representations help disentangle latent factors and boost the networks performance (Higgins et al., 2016). Empirically, we found that putting a sparse (L1) regularizer L L1 = 1 NL X n;”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “One recent paper (Higgins et al., 2016) uses an objective function that seeks representations of statistically independent factors in images, by introducing a regularization term that pushes the distribution of latent factors learned in a generative model to be close to a unit Gaussian.”
Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study (2017, background): “Recent pushes have aimed to better understand DNNs: tailor-made loss functions and architectures produce more interpretable features (Higgins et al., 2016; Raposo et al., 2017) while output-behavior analyses unveil previously opaque operations of these networks (Karpathy et al., 2015).”
An Information-Theoretic Analysis of Deep Latent-Variable Models (2017, background): “In Higgins et al. (2016) the authors motivate the β -VAE objective from a combined neuroscience and information theoretic perspective.”
Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders (2016, methodology): “Inspired by the learning pressure in the ventral visual stream, Higgins et al. (2016) were able to extract disentangled features from images by adding a regularisation coefﬁcient to the lower bound of the VAE.”
Neuroscience-Inspired Artificial Intelligence. (2017, background): “…constructive nature, with humans able to construct ﬁctitious mental scenarios by recombining familiar elements in novel ways, necessitating compositional/disentangled representations of the form present in certain generative models (Eslami et al., 2016; Higgins et al., 2016; Rezende et al., 2016a).”