Disentangling Factors of Variation via Generative Entangling

By Guillaume Desjardins, Aaron C. Courville, Yoshua Bengio. 2012

105 citations recorded by the source corpus.

Here we propose a novel model family with the objective of learning to disentangle the factors of variation in data. Our approach is based on the spike-and-slab restricted Boltzmann machine which we generalize to include higher-order interactions among multiple latent variables. Seen from a generative perspective, the multiplicative interactions emulates the entangling of factors of variation. Inference in the model can be seen as disentangling these generative factors. Unlike previous attempts at disentangling latent factors, the proposed model is trained using no supervised information regarding the latent factors. We apply our model to the task of facial expression classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “To the best of our knowledge, the only other unsupervised method that learns disentangled representations is hossRBM [13], a higher-order extension of the spike-and-slab restricted Boltzmann machine that can disentangle emotion from identity on the Toronto Face Dataset [22].”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The lack of almost any identifiability result in non-linear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2019) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen and Welling, 2014b).”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “, when the factors of variation are not known in advance [9, 32, 45, 7, 25, 30, 6].”
Deep Convolutional Inverse Graphics Network (2015, neutral): “Caption: In order to directly enforce invariance of the latents corresponding to properties of the image which do not change within a given batch, we calculate gradients for the z_k != z_1 which move them towards the mean of each invariant latent over the batch.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Disentangled representations in deep generative models: Disentanglement of generative models have long been studied (Schmidhuber, 1992; Desjardins et al., 2012).”
Generalization in anti-causal learning (2018, background): “Desjardins et al. (2012) successfully disentangle factors through inference over a (causal) generative model of the data.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “, 2018) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen & Welling, 2015).”
Disentangling by Factorising (2018, methodology): “…disentangling include (Schmidhuber, 1992) which attempts to disentangle codes in an autoencoder by penalising predictability of one latent dimension given the others and (Desjardins et al., 2012) where a variant of a Boltzmann Machine is used to disentangle two factors of variation in the data.”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “[11] made an early attempt at unsupervised disentanglement learning by using a higher-order spike-and-slab RBM with block-sparse connectivity to model the multiplicative interactions between (unknown) factors of variation.”
Towards a Definition of Disentangled Representations (2018, background): “A number of purely unsupervised approaches to disentangled factor learning have been proposed [67, 24, 74], however they did not tend to scale well beyond toy datasets.”
Isolating Sources of Disentanglement in VAEs (2018, background): “Prior to β-VAE [7], some have shown successful disentanglement in limited settings with few factors of variation [1, 14, 30, 31].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “It is a challenging problem and many of the earlier attempts have not been able to scale well for realistic settings (Schmidhuber, 1992; Desjardins et al., 2012; Cohen and Welling, 2015) (see also, Higgins et al. (2017)).”
Understanding disentangling in β-VAE (2018, background): “A number of purely unsupervised approaches to disentangled factor learning have been proposed [36, 10, 39, 8, 9, 6, 15], including β -VAE [15], the focus of this text. β -VAE is a state of the art model for unsupervised visual disentangled representation learning.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…such a factorised description of the world is addressed by the large body of literature on disentangled representation learning (Schmidhuber, 1992; Desjardins et al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et…”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Many prior works have also focused on learning controllable and disentangled representations [41, 5, 6, 38, 7, 46].”
Deep Learning of Representations: Looking Forward (2013, methodology): “…priors on the form of the model can be used to enhance disentangling, such as multiplicative interactions between the factors (Tenenbaum and Freeman, 2000; Desjardins et al., 2012) or orthogonality of the features derivative with respect to the input (Rifai et al., 2011b, 2012a; Sohn et al., 2013).”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, neutral): “This example comes from [13].”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “We can edit the set of features that models have access to by ablating sparse features that humans deem spurious; we find that this is significantly more effective than a neuron-based ablation method which has an unfair advantage.”
A Framework for the Quantitative Evaluation of Disentangled Representations (2018, methodology): “As argued in several notable works (Desjardins et al., 2012; Bengio et al., 2013; Chen et al., 2016; Higgins et al., 2017), this understanding can only be gained if the model learns to disentangle the underlying explanatory factors hidden in unlabelled input.”