Structured Disentangled Representations

By Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N. Siddharth, Brooks Paige, Dana H. Brooks, Jennifer G. Dy, Jan-Willem van de Meent. 2018

183 citations recorded by the source corpus.

Deep latent-variable models learn representations of high-dimensional data in an unsupervised manner. A number of recent efforts have focused on learning representations that disentangle statistically independent axes of variation by introducing modifications to the standard objective function. These approaches generally assume a simple diagonal Gaussian prior and as a result are not able to reliably disentangle discrete factors of variation. We propose a two-level hierarchical objective to control relative degree of statistical independence between blocks of variables and individual variables within blocks. We derive this objective as a generalization of the evidence lower bound, which allows us to explicitly represent the trade-offs between mutual information between data and representation, KL divergence between representation and prior, and coverage of the support of the empirical data distribution. Experiments on a variety of datasets demonstrate that our objective can not only disentangle discrete variables, but that doing so also improves disentanglement of other variables and, importantly, generalization even to unseen combinations of factors.

Subfields: Artificial Intelligence, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Flexibly Fair Representation Learning by Disentanglement (2019, background): “Chen et al. (2018); Kim & Mnih (2018) and Esmaeili et al. (2019) argue that decomposing the ELBO and penalizing the total correlation increases disentanglement in the latent representations.”
On the Binding Problem in Artificial Neural Networks (2020, background): “A useful approach is to measure how well the system copes with particular generalization regimes such as to held-out-combinations of features for disentanglement (Esmaeili et al., 2019) and separation (Santoro et al.”
Weakly Supervised Disentanglement with Guarantees (2019, background): “…disentanglement research has focused on the learning of such representations in an unsupervised fashion, using only independent samples from the data distribution without access to the true factors of variation (Higgins et al., 2017; Chen et al., 2018a; Kim & Mnih, 2018; Esmaeili et al., 2018).”
On the Fairness of Disentangled Representations (2019, background): “For the representation to be disentangled, the loss is enriched with a regularizer that encourages structure in the aggregate encoder distribution [2, 14, 13, 24, 36, 49, 65].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “representation that captures this product space is expected to help in generalizing systematically in this regard [18, 22, 59].”
Towards a Definition of Disentangled Representations (2018, background): “Most of the recent approaches to disentangled representation learning [39, 15, 47, 13, 51, 7, 28], as well as the majority of the disentanglement metrics [39, 15, 47, 27] assume that this property should be true.”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “Starting from the decomposition (7), Esmaeili et al. [8] hierarchically decompose the D KL -term in (7) into a regularization term of the dependencies between groups of latent variables G = { G k } n G k =1 and regularization of the dependencies between the random variables in each group G k .”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, background): “The most prominent of such models are β-VAEs and their extensions [4, 9, 8, 7, 15, 5], in which the authors introduce adjustable hyperparameters in the VAE objective to encourage disentanglement.”
SOM-VAE: Interpretable Discrete Representation Learning on Time Series (2019, background): “A lot of work has been done to improve them in this regard, in GANs (Chen et al., 2016) as well as VAEs (Higgins et al., 2017; Esmaeili et al., 2018).”
Unsupervised Object Segmentation by Redrawing (2019, methodology): “In a similar fashion, HFVAE [15] introduces a hierarchical decomposition of the KL term to impose a structure on the latent space.”
Hyperprior Induced Unsupervised Disentanglement of Latent Representations (2018, methodology): “With advancements in reparameterization for discrete distributions, recent work (Esmaeili et al. 2018; Pineau and Lelarge 2018) have proposed adding different priors to different subsets of the latent code to separately model discrete and continuous factors of variation in the data.”
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions (2019, background): “This approach has been very successful in generating representations in deep learning that can be manipulated while creating realistic inputs [3, 4, 8, 16, 25].”
Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis (2020, background): “The most similar hierarchical decomposition to our approach is proposed by [31], but the intention of this decomposition is to facilitate learning statistically independent random variables.”
A Spectral Regularizer for Unsupervised Disentanglement (2018, methodology): “…method induces disentangled representations in GANs of quality comparable to those obtained by VAE-based approaches (Higgins et al. (2016), Kim & Mnih (2018), Chen et al. (2018), Esmaeili et al. (2018)), without the need to introduce auxiliary models into the training procedure (Chen et al., 2016).”
MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders (2019, background): “Mutual information inspired objectives have been explored in GANs (Goodfellow et al., 2014; Chen et al., 2016), clustering (Hinton et al., 1995; Krause et al., 2010) and representation learning (Esmaeili et al., 2018; Hjelm et al., 2018).”
Disentangling and Learning Robust Representations with Natural Clustering (2019, background): “Disentangled representations, defined as those where each latent variable is responsible for only one generative factor in the data while being relatively invariant to other factors [1], allow for more easily interpretable models, [2], and facilitate generalization to previously unseen combinations ”