Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

By Aapo Hyvärinen, Hiroaki Sasaki, Richard E. Turner. 2018

423 citations recorded by the source corpus.

Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other available information. We propose to learn nonlinear ICA by discriminating between true augmented data, or data in which the auxiliary variable has been randomized. This enables the framework to be implemented algorithmically through logistic regression, possibly in a neural network. We provide a comprehensive proof of the identifiability of the model as well as the consistency of our estimation method. The approach not only provides a general theoretical framework combining and generalizing previously proposed nonlinear ICA models and algorithms, but also brings practical advantages.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “…the non-linear ICA community recently shifted to non-iid data types exploiting time dependent or grouped observations (Hyvarinen and Morioka, 2016; Hyvarinen et al., 2019; Gresele et al., 2019) We focus our study on the setting where factors of variations are not observable at all, that is, we…”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “Our contribution is based on recent theoretical results in the nonlinear ICA literature (Hyvärinen et al., 2019; Khemakhem et al., 2020a,b) that assume the data is explained by unobserved and meaningful latent variables, or factors, Z, which are transformed by a decoder, or mixing function, f , to…”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Ideas related to disentangling the factors of variation date back to the non-linear ICA literature (Comon, 1994; Hyv¨arinen and Pajunen, 1999; Bach and Jordan, 2002; Jutten and Karhunen, 2003; Hyvarinen and Morioka, 2016; Hyvarinen et al., 2019; Gresele et al., 2019).”
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series (2020, methodology): “As in Hyv¨arinen et al. (2019), u is some auxiliary variable, and iVAE provides a novel algorithm to estimate nonlinear independent components in the same identiﬁ-able framework. iVAE however suffers from the same problems as TCL, as its auxiliary variable u has to be observed.”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “Another interesting direction to address identi-ﬁability of deep generative model is non-linear ICA, but typically requires observation of auxiliary variables (Hyvarinen, Sasaki, and Turner 2019; Khemakhem et al. 2020).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Recently, the ICA community has considered weak forms of supervision such as temporal consistency (Hyvarinen & Morioka, 2016; 2017), auxiliary supervised information (Hyvarinen et al., 2019; Khemakhem et al., 2019), and multiple views (Gresele et al., 2019).”
On the Fairness of Disentangled Representations (2019, background): “Ideas related to disentangling the factors of variations have a long tradition in machine learning, dating back to the non-linear ICA literature [17, 3, 46, 42, 43, 44, 32].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “…prior, using λ 1 = 0 and λ 2 ≥ 1 with R 2 ( q φ ( z ) Other related works Learning disentangled representations is similar in spirit to non-linear ICA, although it relies primarily on (architectural) inductive biases and different degrees of supervision [13, 2, 39, 36, 37, 38, 25, 33, 32].”
The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA (2019, methodology): “This is closely related to the Assumption of Variability in [23].”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “, 2016), and Independent Component Analysis models (Hyvärinen et al., 2001; Hyvärinen et al., 2018), are built on this assumption.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “The lack of almost any identifiability result in nonlinear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2018) and partially motivated alternative machine learning approaches (Desjardins et al.”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “A similar assumption on the underlying model is likewise a key requirement for the recent extension of identificability results of non-linear ICA (Hyvarinen et al., 2018).”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, methodology): “Recent work in nonlinear Independent Component Analysis (ICA) theory (Hyvärinen and Morioka, 2016, 2017; Hyvärinen et al., 2019) provided the first identifiability results for deep latent-variable models.”
Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, methodology): “Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes.”
Disentangling Factors of Variation Using Few Labels (2019, background): “[28] Aapo Hyvarinen, Hiroaki Sasaki, and Richard E Turner.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “This is a very common assumption in the related literature [39, 54, 57, 58, 63, 69, 82, 83, 129] that may be relaxed in future work.”
Disentangling Factors of Variations Using Few Labels (2020, background): “This problem has already been studied extensively in the non-linear ICA literature (Comon, 1994; Bach & Jordan, 2002; Jutten & Karhunen, 2003; Hyvarinen & Morioka, 2016; Hyvärinen & Pajunen, 1999; Hyvarinen et al., 2019; Gresele et al., 2019).”
On Linear Identifiability of Learned Representations (2020, methodology): “It should be noted that Theorem 1 in (Hyvärinen et al., 2018) provides a potential avenue for further generalization of our Theorem 1 to discriminative models with non-linear interaction between f and g.”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “Recently, Hyvärinen and colleagues [12, 13, 42] showed that a solution to the non-identiﬁability of nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself.”
Relative gradient optimization of the Jacobian term in unsupervised deep learning (2020, background): “Square and randomly initialized (e.g. with uniform or normally distributed entries) weight matrices are known to be invertible with probability one; invertibility of the weight matrices throughout the training is guaranteed by the fact that the L2J terms would diverge for singular matrices (though h”