Variational Autoencoders and Nonlinear ICA: A Unifying Framework

By Ilyes Khemakhem, Diederik P. Kingma, Aapo Hyvärinen. 2019

785 citations recorded by the source corpus.

The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the data. Often, we're interested in going a step further, and want to approximate the true joint distribution over observed and latent variables, including the true prior and posterior distributions over latent variables. This is known to be generally impossible due to unidentifiability of the model. We address this issue by showing that for a broad family of deep latent-variable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Our result requires a factorized prior distribution over the latent variables that is conditioned on an additionally observed variable, such as a class label or almost any other observation. We build on recent developments in nonlinear ICA, which we extend to the case with noisy, undercomplete or discrete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…interesting given recent identifiability results in non-linear ICA (Hyvarinen and Morioka, 2016) that enable semi-supervised Sorrenson et al. (2020); Khemakhem et al. (2020) and weakly-supervised approaches Bouchacourt et al. (2018); Hosoya (2019); Shu et al. (2020); Locatello et al. (2020a).”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “(2) Variational approximation to an unnormalized target density p [34, 59, 33, 30, 28], based on min α KL( q α p ) , where KL( q α p ) is the difference between energy and entropy, i.e., we want q α to have low energy but high entropy.”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “An estimation procedure which relies on variational autoencoders (VAEs) (Kingma and Welling, 2014) and learned causal mechanisms regularized for sparsity via binary masks.”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “Recent work combines non-linear ICA with disentanglement (Khemakhem et al., 2020; Sorrenson et al., 2020; Klindt et al., 2020).”
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series (2020, methodology): “Recent theoretical and practical advances in Nonlinear ICA provide a principled approach to this problem (Hyv¨arinen and Morioka, 2016; Hyv¨arinen and Morioka, 2017; Hyv¨arinen et al., 2019; Khemakhem et al., 2020; Sorrenson et al., 2020).”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “Another interesting direction to address identifiability of deep generative model is non-linear ICA, but typically requires observation of auxiliary variables (Hyvarinen, Sasaki, and Turner 2019; Khemakhem et al. 2020).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “Recently, the ICA community has considered weak forms of supervision such as temporal consistency (Hyvarinen & Morioka, 2016; 2017), auxiliary supervised information (Hyvarinen et al., 2019; Khemakhem et al., 2019), and multiple views (Gresele et al.”
Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, methodology): “A more general formulation, valid for other forms of data, was given in Hyv¨arinen et al. (2018) and the theory was extended to generative models in Khemakhem et al. (2019), where experiments were implemented by a VAE.”
Disentanglement through Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, background): “1 EXISTING THEORY OF NONLINEAR ICA This section adapts theoretical results from Khemakhem et al. (2019) to the context of invertible neural networks.”
Contrastive Learning Inverts the Data Generating Process (2021, background): “Recent work has generalized this contribution to VAEs (Khemakhem et al., 2020a; Locatello et al., 2020; Klindt et al., 2021), as well as (invertible-by-construction) energy-based models (Khemakhem et al., 2020b).”
A Cookbook of Self-Supervised Learning (2023, background): “…that InfoNCE balances alignment of “positive” examples and uniformity of the overall feature representation [Wang and Isola, 2022], or that (under strong assumptions) it can identify the latent structure in a hypothesized data-generating process, akin to nonlinear ICA [Khemakhem et al., 2020].”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, background): “Flow-based models have gained popularity in generative modeling [9, 10, 31, 18, 3, 35, 53] and variational inference [30, 49, 32, 28, 26].”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “This is a very common assumption in the related literature [39, 54, 57, 58, 63, 69, 82, 83, 129] that may be relaxed in future work.”
On Linear Identifiability of Learned Representations (2020, methodology): “This requirement is similar to the conditions in earlier work on nonlinear ICA such as [Hyvärinen et al., 2018], as discussed in Section 6.”
CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models (2020, methodology): “We adopt the ∼ - identiﬁable [11] as follows: Deﬁnition 1 Let ∼ be the binary relation on Θ deﬁned as follows: where C = ( I − A T ) − 1 .”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “We build on previous theory [9, 10] to provide a novel identifiability proof for our model under mild conditions.”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “Causal representation learning Finally, the ideas here connect to causal representation learning (e.g., Higgins et al., 2016; Hyvarinen & Morioka, 2016; Higgins et al., 2018; Khemakhem et al., 2020; Zimmermann et al., 2021; Sch¨olkopf et al., 2021; Moran et al., 2021; Wang et al., 2023).”
Dual Adversarial Inference for Text-to-Image Synthesis (2019, background): “Note that an alternative approach for ICA has also recently been explored in [13].”