Variational Inference of Disentangled Latent Concepts from Unlabeled Observations

By Abhishek Kumar, Prasanna Sattigeri, Avinash Balakrishnan. 2017

602 citations recorded by the source corpus.

Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We also propose a new disentanglement metric which is better aligned with the qualitative disentanglement observed in the decoder's output. We empirically observe significant improvement over existing methods in terms of both disentanglement and data likelihood (reconstruction quality).

Subfields: Artificial Intelligence, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “The scores proposed by Eastwood and Williams (2018); Kumar et al. (2018); Chen et al. (2018); Ridgeway and Mozer (2018) first construct a matrix of relation between factors of variation and codes (for example pairwise mutual information) and then aggregate this matrix into a single final number.”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “…to p ( z ), diﬀerent regularization strategies have been proposed, along with evaluation metrics to gauge the disentanglement of the learned representations (Higgins et al., 2017a; Kim and Mnih, 2018; Burgess et al., 2018; Kumar et al., 2018; Chen et al., 2018; Eastwood and Williams, 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Recently, there has been a renewed interest in the unsupervised learning of disentangled representations (Higgins et al., 2017a; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018) along with quantitative evaluation (Kim & Mnih, 2018; Eastwood & Williams, 2018; Kumar et al”
Weakly Supervised Disentanglement with Guarantees (2019, neutral): “This is in contrast to the existing disentanglement definitions and metrics, which require access to the ground truth factors (Higgins et al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018a; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “They should allow to integrate out nuisance factors [22], to perform interventions and to answer counterfactual questions [39, 44, 40].”
On the Fairness of Disentangled Representations (2019, methodology): “First, we take the 12 600 pre-trained models of [61], which cover a large number of hyperparameters and random seeds for the most prominent approaches: -VAE, AnnealedVAE, Factor-VAE, -TCVAE, DIP-VAE-I and II.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “Similarly for DIP-VAE we consider both the DIP-VAE-I and DIP-VAE-II variations of the proposed regularizer [47].”
Generalization in anti-causal learning (2018, background): “Based on the previous paragraph, it makes sense that most state-of-the-art methods for disentangling are based on autoencoder architectures (Higgins et al., 2017; Kim and Mnih, 2018; Kumar et al., 2017).”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, methodology): “…made, either with some sort of supervision (e.g. Siddharth et al., 2017; Bouchacourt et al., 2017; Liu et al., 2017; Mathieu et al., 2016; Cheung et al., 2014) or completely unsupervised (e.g. Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018; Es-maeili et al., 2018).”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, neutral): “43–45 Uncertainty Quantiﬁcation .”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “…to a factorial p ( z ) [7, 16], forcing the variational aggregated prior q ( z ) to be similar to the prior p ( z ) [28], adding total correlation loss [18], forcing the covariance matrix of q ( z ) to be close to the identity matrix [23], and using a kernel-based measure of independence [27].”
Towards a Definition of Disentangled Representations (2018, background): “To date, most work on disentangling has relied on comparison with human intuition for evaluation, though a number of metrics have been proposed in the case where the ground truth factors are known [27, 65, 51, 47, 55, 72].”
Isolating Sources of Disentanglement in VAEs (2018, background): “As a means to describe the properties of disentangled representations, factorial representations have been motivated by many [1, 2, 3, 22, 32, 33].”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “echanisms for the GRM, beneting on many downstream tasks across domains under the lifelong learning framework. The approach proposed in this paper also addresses disentangled representation learning [35] in the context of lifelong learning. Many recent approaches would aim to modify the VAE framewor”
On Variational Bounds of Mutual Information (2019, background): “Many recent papers in representation learning have focused on learning latent representations in a generative model that correspond to human-interpretable or “disentangled” concepts (Higgins et al., 2016; Burgess et al., 2018; Chen et al., 2018; Kumar et al., 2017).”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Disentanglement β -VAE (6) [2], FactorVAE (8) [3], β -TCVAE (9) [4], InfoVAE (9) [5], DIP-VAE (11) [6], HSIC-VAE (12) [7], HFVAE (13) [8], VIB [9], Information dropout (15) [10], DC-IGN [11], FaderNetworks (18) [12], VFAE (17) [13] Hierarchical representation 1 PixelVAE [14], LVAE [15], VLaAE [16],…”
Structured Disentangled Representations (2018, methodology): “This has prompted a number of approaches that modify the VAE objective by adding, removing, or altering the weight of individual terms [Kumar et al., 2017, Zhao et al., 2017, Gao et al., 2018, Achille and Soatto, 2018].”