Discovering Hidden Factors of Variation in Deep Networks

By Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, Barbara Olshausen. 2014

199 citations recorded by the source corpus.

Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject identity in faces. We demonstrate this on the MNIST handwritten digit database, the Toronto Faces Database (TFD) and the Multi-PIE dataset by generating manipulated instances of the data. Furthermore, we demonstrate these deep networks can extrapolate `hidden' variation in the supervised signal.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, neutral): “The quality of our unsupervised disentangled representation matches previous works that made use of supervised label information [5–9].”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017; Suter et al., 2019).”
Adversarial Autoencoders (2015, background): “Importantly, we demonstrated how it can be used to disentangle the style and label information of a dataset (Kingma et al., 2014; Cheung et al., 2014).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Initially, the community focused on more or less explicit and task dependent supervision (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[9] Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen.”
On the Fairness of Disentangled Representations (2019, background): “[77, 15, 66, 70, 50, 52, 1] assume partially observed factors of variation that should be disentangled from the other unobserved ones.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “Given that unsupervised algorithms did not initially perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Mathieu et al., 2016; Na”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “Various proposals for more structure imposing regularization have been made, either with some sort of supervision (e.g. Siddharth et al., 2017; Bouchacourt et al., 2017; Liu et al., 2017; Mathieu et al., 2016; Cheung et al., 2014) or completely unsupervised (e.”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Isolating Sources of Disentanglement in VAEs (2018, background): “The goal in this case is not perfect inversion but to distill simpler representation [15, 25, 26, 27, 28].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “, 2017), (iv) knowledge of a complementary signal to infer representations that are conditionally independent of it(1) (Cheung et al., 2014; Mathieu et al., 2016; Siddharth et al., 2017).”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et al., 2015; Goroshin et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Whitney et al., 2016; Karaletsos et al., 2016; Chen et al., 2016; Higgins et al., 2017).”
Disentangling factors of variation in deep representation using adversarial training (2016, background): “In [2], the authors show that standard deep architectures can discover and explicitly represent factors of variation aside those relevant for classiﬁcation, by combining autoencoders with simple regularization terms during the training.”
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis (2015, background): “In [6], the (potentially unknown) latent factors of variation are both discovered and disentangled using a novel hidden unit regularizer.”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Many prior works have also focused on learning controllable and disentangled representations [41, 5, 6, 38, 7, 46].”
Bayesian representation learning with oracle constraints (2015, methodology): “In Cheung et al. (2014), deep generative models are used with functional constraints on the latent space to increase speciﬁcity of latent variables, which is a goal we share but tackle using the oracle-information as a model-based semantic regularizer.”
View Synthesis by Appearance Flow (2016, background): “[13] proposed an auto-encoder with decoupled semantic units representing pose, identity etc.”