Learning Disentangled Representations with Semi-Supervised Deep Generative Models

By Siddharth Narayanaswamy, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank D. Wood, Philip H. S. Torr. 2017

378 citations recorded by the source corpus.

Variational autoencoders (VAEs) learn representations of data by jointly training a probabilistic encoder and decoder network. Typically these models encode all features of the data into a single variable. Here we are interested in learning disentangled representations that encode distinct aspects of the data into separate variables. We propose to learn such representations using model architectures that generalise from standard VAEs, employing a general graphical model structure in the encoder and decoder. This allows us to train partially-specified models that make relatively strong assumptions about a subset of interpretable variables and rely on the flexibility of neural networks to learn representations for the remaining variables. We further define a general objective for semi-supervised learning in this model class, which can be approximated using an importance sampling procedure. We evaluate our framework's ability to learn disentangled representations, both by qualitative exploration of its generative capacity, and quantitative evaluation of its discriminative ability on a variety of models and datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Sometimes generative models are also evaluated using linear classiﬁcation (Radford et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), or semi-supervised learning (Kingma et al., 2014; Narayanaswamy et al., 2017; Tschannen et al., 2018).”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “Given that unsupervised algorithms did not initially perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Na”
Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro (2019, methodology): “Within the Python community, some examples of embedded PPLs include Pyro [2] and ProbTorch [3] based on PyTorch [4], TensorFlow Probability [5] and Edward2 [6] based on TensorFlow, and PyMC3 [7] based on Theano.”
Pyro: Deep Universal Probabilistic Programming (2018, background): “ProbTorch (Siddharth et al. (2017)) is a PPL built on PyTorch with a focus on deep generative models and developing new objectives for variational inference.”
An Introduction to Probabilistic Programming (2018, background): “On top of PyTorch, at time of writing a similar distributions library is in development, and two probabilistic programming languages (including Pyro (Uber, 2018) and probabilistic Torch (Siddharth et al., 2017)) are built.”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “We evaluate V ERSA by comparing it to a conditional variational autoencoder (C-VAE) with view angles as labels (Kingma et al., 2014; Narayanaswamy et al., 2017) and identical architectures.”
Toward Controlled Generation of Text (2017, background): “Siddharth et al. (2017); Kingma et al. (2014) base on VAEs and obtain disentangled image representations with semi-supervised learning.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Initially, the community focused on more or less explicit and task dependent supervision (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017).”
Weakly Supervised Disentanglement with Guarantees (2019, background): “This form of supervision is often leveraged in style-content disentanglement, where labels are available for content but not style (Kingma et al., 2014; Narayanaswamy et al., 2017; Chen et al., 2018b; Gabbay & Hoshen, 2019).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[43] Siddharth Narayanaswamy, T Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr.”
On the Fairness of Disentangled Representations (2019, background): “[77, 15, 66, 70, 50, 52, 1] assume partially observed factors of variation that should be disentangled from the other unobserved ones.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “approaches [14, 46, 11]. Given that unsupervised algorithms did not initially perform well on realistic settings most of the other works have considered some more or less explicit form of supervision [42, 53, 29, 26, 9, 35, 37, 55, 49]. [20, 10] assume some knowledge of the effect of the factors of ”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “Various proposals for more structure imposing regularization have been made, either with some sort of supervision (e.g. Siddharth et al., 2017; Bouchacourt et al., 2017; Liu et al., 2017; Mathieu et al., 2016; Cheung et al., 2014) or completely unsupervised (e.g. Higgins et al., 2017; Kim & Mnih,…”
Disentangling by Factorising (2018, background): “…or explicit knowledge about the true underlying factors of the data have excelled at disentangling (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Siddharth et al., 2017; Hinton et al., 2011; Mathieu et al., 2016; Goroshin et al., 2015; Hsu et al., 2017; Denton & Birodkar, 2017).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “[40] replace the variational objective of the M2 model with the importance-weighted loss [6] so that q φ ( y, z | x ) can have arbitrary conditional dependency between y and z instead of just the decomposition q φ ( y, z | x ) = q φ ( y | x ) q φ ( z | x, y ) used in [21].”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, background): “…have focused on the use of neural networks to learn independent factors of variation in static datasets – through the encouragement of disentangled representations (Narayanaswamy et al., 2017; Kim & Mnih, 2018) or by learning causal mechanisms (Parascandolo et al., 2018; Thomas et al., 2018).”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “, 2017), (iv) knowledge of a complementary signal to infer representations that are conditionally independent of it(1) (Cheung et al., 2014; Mathieu et al., 2016; Siddharth et al., 2017).”