Information Constraints on Auto-Encoding Variational Bayes

By Romain Lopez, Jeffrey C. Regier, Nir Yosef, Michael I. Jordan. 2018

156 citations recorded by the source corpus.

Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on Auto-Encoding Variational Bayes and whose search space is constrained via kernel-based measures of independence. In particular, our method employs the $d$-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how to apply this method to a range of problems, including the problems of learning invariant representations and the learning of interpretable representations. We also present a full-fledged application to single-cell RNA sequencing (scRNA-seq). In this setting the biological signal is mixed in complex ways with sequencing errors and sampling effects. We show that our method out-performs the state-of-the-art in this domain.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangled representations have been found to be more sample-efficient [29], less sensitive to nuisance variables [55], and better in terms of (systematic) generalization [1, 16, 28, 35, 70].”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “…to a factorial p ( z ) [7, 16], forcing the variational aggregated prior q ( z ) to be similar to the prior p ( z ) [28], adding total correlation loss [18], forcing the covariance matrix of q ( z ) to be close to the identity matrix [23], and using a kernel-based measure of independence [27].”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “Lopez et al. [7] also consider the problem of censoring side information, but use the HSIC regularizer instead of MMD.”
Disentangling and Learning Robust Representations with Natural Clustering (2019, background): “te posterior q(z)=E p(x)[q(zjx)] and the factorial prior p(z). This encourages the dimensions of z to become decorrelated while maintaining their sensitivity to the inputs. Esmaeili et al. (2018) and Regier et al. (2018) both propose methods that allow the reduction of dependencies between subgroups”
Mixtures of Variational Autoencoders (2020, background): “The K components are mutually independent if their joint distribution is equal to the tensor product of their marginal distributions, [18]. dHSIC is deﬁned as: where M 2 ( n ) , M 2 K ( n ) and M K +1 ( n ) represent the sums of kernel products for the given n variables.”
Mapping single-cell data to reference atlases by transfer learning (2021, background): “Architectural surgery can extend any conditional neural network-based data-integration method to enable decentralized reference updating, facilitate model reuse and provide a framework for learning from reference data.”
InfoVAEGAN: Learning Joint Interpretable Representations by Information Maximization and Maximum Likelihood (2021, background): “However, TC is biased and is zero only if estimated on the whole dataset, [9].”
Deep Mixture Generative Autoencoders (2021, background): “The K components are mutually independent if their joint distribution is equal to the product of their marginal distributions, [39].”
Invariant Representations through Adversarial Forgetting (2019, methodology): “The Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al. 2005) constrained VAE (HCV) (Lopez et al. 2018) uses HSIC to enforce independence between the latent embedding and s .”
Weakly Supervised Disentanglement by Pairwise Similarities (2019, methodology): “In recent years, several methods (Higgins et al. 2017; Kim and Mnih 2018; Chen et al. 2018; Lopez et al. 2018) have been proposed to solve disentanglement learning under the Variational Autoencoder (VAE) framework.”