Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)

By Peter Sorrenson, Carsten Rother, Ullrich Köthe. 2020

137 citations recorded by the source corpus.

A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressible-flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…to be particularly interesting given recent identifiability results in non-linear ICA (Hyvarinen and Morioka, 2016) that enable semi-supervised Sorrenson et al. (2020); Khemakhem et al. (2020) and weakly-supervised approaches Bouchacourt et al. (2018); Hosoya (2019); Shu et al. (2020);…”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “, 2021), normalizing flows (Sorrenson et al., 2020) and energy-based models (Khemakhem et al.”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “Recent work combines non-linear ICA with disentanglement (Khemakhem et al., 2020; Sorrenson et al., 2020; Klindt et al., 2020).”
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series (2020, methodology): “For linear ICA this approach has been shown to yield identiﬁable models (Belouchrani et al., 1997; Tong et al., 1991), and extensions to the nonlinear case have been also been proposed in earlier work (Harmel-ing et al., 2003; Sprekeler et al., 2014).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “…of disentangled representations is theoretically impossible without inductive biases, the focus shifted back to semi-supervised (Locatello et al., 2020; Sorrenson et al., 2020; Khemakhem et al., 2019) and weakly-supervised approaches (Bouchacourt et al., 2018; Hosoya, 2019; Shu et al., 2020).”
On Linear Identifiability of Learned Representations (2020, background): “Our results extend recent advances in the theory of nonlinear Independent Components Analysis (ICA), which have recently provided strong identifiability results for generative models of data (Hyvärinen et al., 2018; Khemakhem et al., 2019; 2020; Sorrenson et al., 2020).”
iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression (2021, methodology): “The proposed iVPF is constructed from volume preserving flow [7, 34] to achieve the bijective mapping on discrete space, and an efficient lossless compression algorithm based on iVPF is developed.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “This development has sparked a renewed interest in identifiability in the context of deep representation learning [63, 64, 69, 83, 102, 108, 109, 129].”