Learning Factorial Codes by Predictability Minimization

By J. Urgen Schmidhuber. 1992

I propose a novel general principle for unsupervised learning of distributed nonredundant internal representations of input patterns. The principle is based on two opposing forces. For each representational unit there is an adaptive predictor, which tries to predict the unit from the remaining units. In turn, each unit tries to react to the environment such that it minimizes its predictability. This encourages each unit to filter "abstract concepts" out of the environmental input such that these concepts are statistically independent of those on which the other units focus. I discuss various simple yet potentially powerful implementations of the principle that aim at finding binary factorial codes (Barlow et al. 1989), i.e., codes where the probability of the occurrence of a particular input is simply the product of the probabilities of the corresponding code symbols. Such codes are potentially relevant for (1) segmentation tasks, (2) speeding up supervised learning, and (3) novelty detection. Methods for finding factorial codes automatically implement Occam's razor for finding codes using a minimal number of units. Unlike previous methods the novel principle has a potential for removing not only linear but also nonlinear output redundancy. Illustrative experiments show that algorithms based on the principle of predictability minimization are practically feasible. The final part of this paper describes an entirely local algorithm that has a potential for learning unique representations of extended input sequences.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Generative Adversarial Nets (2014, background): “The most relevant work is predictability minimization [26].”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The lack of almost any identifiability result in non-linear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2019) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen and Welling, 2014b).”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “There are various definitions for disentanglement [50, 46, 1, 6, 18], but a common goal is a latent space that consists of linear subspaces, each of which controls one factor of variation.”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “A similar objective in (Schmidhuber, 1992) learns a two-way mapping to a representation with marginally independent units.”
Learning deep representations by mutual information estimation and maximization (2018, background): “…can be compact (Gretton et al., 2012), independent (Hyv ¨ arinen & Oja, 2000; Hinton, 2002; Dinh et al., 2014; Brakel & Bengio, 2017), disentangled (Schmidhuber, 1992; Rifai et al., 2012; Bengio et al., 2013; Chen et al., 2018; Gonzalez-Garcia et al., 2018), or independently controllable (Thomas…”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Disentangled representations in deep generative models: Disentanglement of generative models have long been studied (Schmidhuber, 1992; Desjardins et al., 2012).”
Learning Adversarially Fair and Transferable Representations (2018, background): “, 2016), although similar concepts had previously been proposed for unsupervised and supervised learning (Schmidhuber, 1992; Gutmann & Hyvärinen, 2010).”
Deep learning in neural networks: An overview (2014, methodology): “Other nonlinear methods include Predictability Minimization (PM) (Schmidhuber, 1992c), where nonlinear feature detectors fight nonlinear predictors, trying to become both informative and as unpredictable as possible, and LOCOCODE (Hochreiter and Schmidhuber, 1999), where FMS (Sec.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Alternatively, Bengio (2019) argues for the importance of a low-dimensional ‘conscious state’ (working memory) composed of largely independent units of abstraction that can be selected via attention (perhaps reminiscient of Schmidhuber, 1992b).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “Learning well-disentangled representations of complex sensory data has been identified as one of the key challenges in the quest for AI [1, 40, 26, 2, 42, 24, 47], since they should contain all the information present in the observations in a compact and interpretable structure [1, 21, 6] while bein”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Learning good representations of high-dimensional sensory data is of fundamental importance to Artificial Intelligence [4, 3, 6, 49, 7, 69, 67, 50, 59, 73].”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “In the context of learning interpretable representations [2, 17, 5, 40, 38] it is useful to assume that the data originates from a process with some generating factors.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “, 2018) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen & Welling, 2015).”
Disentangling by Factorising (2018, background): “Schmidhuber (1992) quantifies predictability between the different dimensions of z, using a predictor that is trained to predict zj from z−j .”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “Disentanglement learning holds the key for understanding the world from observations, transferring knowledge across different tasks and domains, generating novel designs, and learning compositional concepts [5, 17, 24, 32, 39].”
Towards a Definition of Disentangled Representations (2018, background): “A number of purely unsupervised approaches to disentangled factor learning have been proposed [67, 24, 74], however they did not tend to scale well beyond toy datasets.”
Isolating Sources of Disentanglement in VAEs (2018, methodology): “While the definition of disentanglement is open to debate, many believe a factorial representation, one with statistically independent variables, is a good starting point [1, 2, 3].”
Catastrophic forgetting and mode collapse in GANs (2020, methodology): “INTRODUCTION GANs [1, 2] are a powerful tool for modeling complex distributions.”