Disentangling factors of variation in deep representation using adversarial training

By Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, Yann LeCun. 2016

521 citations recorded by the source corpus.

We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentaglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017; Suter et al., 2019).”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “Generative Adversarial Networks (GANs) [14, 58] have achieved impressive results in image generation [5, 35], image editing [61], and representation learning [35, 39, 33].”
Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (2017, background): “The most similar work to ours (Mathieu et al., 2016) proposed an architecture that combined variational autoencoder (Kingma et al.”
Compositional Fairness Constraints for Graph Embeddings (2019, methodology): “We build upon the success of recent adversarial approaches to fairness (Edwards & Storkey, 2015), disentanglement (Mathieu et al., 2016), and transfer learning (Madras et al.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Initially, the community focused on more or less explicit and task dependent supervision (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[40] Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun.”
On the Fairness of Disentangled Representations (2019, background): “[77, 15, 66, 70, 50, 52, 1] assume partially observed factors of variation that should be disentangled from the other unobserved ones.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Counterfactuals uncover the modular structure of deep generative models (2018, background): “β-VAEs and other works (Chen et al., 2016; Mathieu et al., 2016; Kulkarni et al., 2015; Higgins et al., 2017) address supervised or unsupervised disentanglement of latent variables related to what we formalize as extrinsic disentanglement of transformations acting on data points.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “Given that unsupervised algorithms did not initially perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Mathieu et al., 2016; Na”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “…regularization have been made, either with some sort of supervision (e.g. Siddharth et al., 2017; Bouchacourt et al., 2017; Liu et al., 2017; Mathieu et al., 2016; Cheung et al., 2014) or completely unsupervised (e.g. Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al.,…”
Disentangling by Factorising (2018, background): “…or explicit knowledge about the true underlying factors of the data have excelled at disentangling (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Siddharth et al., 2017; Hinton et al., 2011; Mathieu et al., 2016; Goroshin et al., 2015; Hsu et al., 2017; Denton & Birodkar, 2017).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “[29] proposed a conditional VAE that models both labeled factors of variation s and other unspeciﬁed latent representations z .”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “, 2017), (iv) knowledge of a complementary signal to infer representations that are conditionally independent of it(1) (Cheung et al., 2014; Mathieu et al., 2016; Siddharth et al., 2017).”
Unsupervised Learning of Disentangled Representations from Video (2017, background): “[18] demonstrate the a loss based on GANs can produce sharper generations than traditional (cid:96) 2 -based losses.”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “sserstein GAN loss [85]. While Larsen et al. [82] investigate the representation learned by their model, the focus of these works is on improving the sample quality of VAE and AAE/WAE. Mathieu et al. [86] rely on a similar setup as [82], but use labels to learn disentangled representations. Cross-do”
Fader Networks: Manipulating Images by Sliding Attributes (2017, background): “It is related to information as in [16].”