Learning Independent Features with Adversarial Nets for Non-linear ICA

By Philemon Brakel, Yoshua Bengio. 2017

104 citations recorded by the source corpus.

Reliable measures of statistical dependence could potentially be useful tools for learning independent features and performing tasks like source separation using Independent Component Analysis (ICA). Unfortunately, many of such measures, like the mutual information, are hard to estimate and optimize directly. We propose to learn independent features with adversarial objectives (Goodfellow et al. 2014, Arjovsky et al. 2017) which optimize such measures implicitly. These objectives compare samples from the joint distribution and the product of the marginals without the need to compute any probability densities. We also propose two methods for obtaining samples from the product of the marginals using either a simple resampling trick or a separate parametric distribution. Our experiments show that this strategy can easily be applied to different types of model architectures and solve both linear and non-linear ICA problems.

Subfields: Analytical Chemistry, Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Learning deep representations by mutual information estimation and maximization (2018, methodology): “A similar trick has been used for learning maximally independent representations for sequential data (Brakel & Bengio, 2017).”
Disentangling by Factorising (2018, methodology): “Brakel et al [4] also use a discriminator to minimise the Jensen-Shannon Divergence between the distribution of codes and the product of its marginals, but use the GAN framework with deterministic encoders and decoders, and only explore their technique in the context of Independent Component Analysi”
Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, background): “there is no temporal or similar structure in the data, the model is seriously unidentifiable (Hyvärinen and Pajunen, 1999), although attempts have been made to estimate it nevertheless, often by minimizing the mutual information of outputs of a neural network (Deco and Obradovic, 1995; Tan et al., 2”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, methodology): “We trained these 4 models on the dataset described above, with M = 40 , L = 1000 , d = 5 and n ∈ [2 , 5] .”
Maximum Entropy Generators for Energy-Based Models (2019, background): “As proposed by Brakel & Bengio (2017) in the context of using a discriminator to minimize statistical dependencies between the outputs of an encoder, the samples from the marginals can be obtained by creating negative examples pairing an X and a Z from different samples of the joint, e.g., by…”
MINE: Mutual Information Neural Estimation (2018, methodology): “This observation can be used to help formulate variational Bayes in terms of implicit distributions (Mescheder et al., 2017) or INFOMAX (Brakel & Bengio, 2017).”
CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models (2020, methodology): “[ 11] gives an identiﬁability of VAE with additional inputs, by leveraging the theory of nonlinear Independent Component Analysis (nonlinear ICA) [2].”
Learning Speaker Representations with Mutual Information (2018, methodology): “In [14], a GAN that minimizes MI using positive and negative samples has been proposed for Independent Component Analysis (ICA).”