A Survey of Inductive Biases for Factorial Representation-Learning

By Karl Ridgeway. 2016

81 citations recorded by the source corpus.

With the resurgence of interest in neural networks, representation learning has re-emerged as a central focus in artificial intelligence. Representation learning refers to the discovery of useful encodings of data that make domain-relevant information explicit. Factorial representations identify underlying independent causal factors of variation in data. A factorial representation is compact and faithful, makes the causal factors explicit, and facilitates human interpretation of data. Factorial representations support a variety of applications, including the generation of novel examples, indexing and search, novelty detection, and transfer learning. This article surveys various constraints that encourage a learning algorithm to discover factorial representations. I dichotomize the constraints in terms of unsupervised and supervised inductive bias. Unsupervised inductive biases exploit assumptions about the environment, such as the statistical distribution of factor coefficients, assumptions about the perturbations a factor should be invariant to (e.g. a representation of an object can be invariant to rotation, translation or scaling), and assumptions about how factors are combined to synthesize an observation. Supervised inductive biases are constraints on the representations based on additional information connected to observations. Supervisory labels come in variety of types, which vary in how strongly they constrain the representation, how many factors are labeled, how many observations are labeled, and whether or not we know the associations between the constraints and the factors they are related to. This survey brings together a wide variety of models that all touch on the problem of learning factorial representations and lays out a framework for comparing these models based on the strengths of the underlying supervised and unsupervised inductive biases.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “There are various definitions for disentanglement [50, 46, 1, 6, 18], but a common goal is a latent space that consists of linear subspaces, each of which controls one factor of variation.”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “Overall, we observe that the trends we observed in Section 5.1 for DCI Disentanglement can be observed also for the other disentanglement scores (with the partial exception of Modularity (Ridgeway, 2016)).”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “Principle Components Analysis (PCA) greedily isolates sources of variance in the data, while Independent Component Analysis (ICA) recovers a factorized representation, see [37] for a recent review.”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “Inspired by [5, 35], we adopt the notion of disentangled representation learning as “a process of decorrelating information in the data into separate informative representations, each of which corresponds to a concept deﬁned by humans ”.”
Isolating Sources of Disentanglement in VAEs (2018, background): “While the definition of disentanglement is open to debate, many believe a factorial representation, one with statistically independent variables, is a good starting point [1, 2, 3].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “Indeed, the importance of learning disentangled representations has been argued in several recent works (Bengio et al., 2013; Lake et al., 2016; Ridgeway, 2016).”
Understanding disentangling in β-VAE (2018, background): “In particular, it has been argued that disentangled representations might be helpful [4, 34].”
Adversarial Latent Autoencoders (2020, background): “Indeed, we recall that a disentangled representation [45, 42, 1] refers to a space consisting of linear subspaces, each of which is responsible for one factor of variation.”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “While the proper expression for disentanglement is a subject of philosophical debate, many believe a factorial representation, one with statistically independent variables, is a good starting point (Schmidhuber, 1992; Ridgeway, 2016; Achille & Soatto, 2017).”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, background): “…a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs [29, 34–36], consolidating ideas from neural computation and machine learning [9], and seeking a principled deﬁnition of disentanglement [4, 32, 37, 38].”