Auto-Encoding Total Correlation Explanation

By Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, Aram Galstyan. 2018

84 citations recorded by the source corpus.

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Isolating Sources of Disentanglement in VAEs (2018, background): “Similarly, [34] arrives at the ELBO from an objective that combines informativeness and the total”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “Many recent approaches would aim to modify the VAE framework in order to learn a meaningful representation of the data by imposing a large penalty on the Kullback-Leibler (KL) divergence term [7,17,25] or on the total correlation latent variables [12,20,28,30].”
Structured Disentangled Representations (2018, methodology): “This has prompted a number of approaches that modify the VAE objective by adding, removing, or altering the weight of individual terms [Kumar et al., 2017, Zhao et al., 2017, Gao et al., 2018, Achille and Soatto, 2018].”
InfoVAEGAN: Learning Joint Interpretable Representations by Information Maximization and Maximum Likelihood (2021, methodology): “The TC penalty was used in various VAE frameworks [8] for inducing disentangled representations.”
Learning joint latent representations based on information maximization (2021, methodology): “These research results can be expanded by adapting other disentangled methods, such as by employing the total correlation [23] in the framework described in this paper in order to induce better-disentangled representations.”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, methodology): “We note that layer-wise training with a related information-theoretic objective has been studied by Ver Steeg & Galstyan (2015) and Gao et al. (2018), which have shown an effective and practical method to compose and analyze hierarchical representations.”
Learning Discrete and Continuous Factors of Data via Alternating Disentanglement (2019, background): “AnchorVAE sets β to a small value for a subset of variables and high value for the rest of the variables but does not dynamically control the information ﬂow (Gao et al., 2018).”
Joint-VAE: Learning Disentangled Joint Continuous and Discrete Representations (2018, methodology): “Recently, several methods have been explored to improve the reconstruction quality without decreasing disentanglement (Burgess et al. (2017); Kim & Mnih (2018); Chen et al. (2018); Gao et al. (2018)).”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “Similarly, Gao et al. (2018) arrives at the ELBO from an objective that simultaneously maximizes informativeness and minimizes the total correlation of latent variables.”
Lifelong Teacher-Student Network Learning (2021, methodology): “In other approaches, the total correlation is used as a regularization term in the objective function for ensuring the disentanglement among categories of characteristics in the feature space [15], [16], [17].”
Hierarchical Disentangled Representations (2018, background): “Three recent papers [Kim and Mnih, 2018; Gao et al., 2018; Chen et al., 2018] all aim to address limitations of the β-VAE by specifically focusing on optimizing the term corresponding to the total correlation, a divergence KL (qφ(z) || ∏ d qφ(zd)); although the approaches differ, they share the comm”