Finding Minimum Entropy Codes

By H. Barlow, T. P. Kaushal, Graeme J. Mitchison. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “Ideally, given an ensemble of input patterns, redundancy reduction through a deep NN will create a factorial code (a code with statistically independent components) of the ensemble (Barlow et al., 1989; Barlow, 1989), to disentangle the unknown factors of variation (e.”
On the Binding Problem in Artificial Neural Networks (2020, background): “A disentangled representation aims to make these attributes explicit by establishing a local correspondence between (independent) factors of variation and features (Barlow et al., 1989; Schmidhuber, 1992c; Higgins et al., 2017a, 2018; Ridgeway and Mozer, 2018).”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Learning good representations of high-dimensional sensory data is of fundamental importance to Artificial Intelligence [4, 3, 6, 49, 7, 69, 67, 50, 59, 73].”
Learning Factorial Codes by Predictability Minimization (1992, methodology): “The focus of this paper, however, is on situations like the ones studied in Barlow et al. (1989): Noise-free environments and sufficient representational capacity in the representational units.”
Neural Expectation Maximization (2017, background): “It has been argued that such representations should be distributed [13, 37] and disentangled [1, 31, 3].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “A central objective of unsupervised learning is to compress the observed data [14, 228].”
Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling (1998, neutral): “This is similar to the visual coding theory (Barlow et al., 1989; Field, 1989) which suggests that the sparse coding scheme has advantages over the compact coding scheme.”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, background): “Since factors are independent, any statistical dependence found between factors in a new dataset indicates a new situation [5].”
Autoencoder for words (2014, methodology): “In [12] Barlow wrote, “a representation whose elements are independent makes it possible to form associations with logical functions of the elements, not just with the elements themselves.”
Non linear neurons in the low noise limit : a factorial code maximizes information transferJean (1994, background): “Still, there are works on the optimization of the transfer function[5], on the study of input distributions to which a given transfer function is optimally adapted[21]; on the use of redundancy reduction for binary, more generally discrete, coding[22, 23]; on networks of binary neurons studied with ”
Competition and Multiple Cause Models (1995, background): “One approach that has been tried to counter this is to add terms to the objective function encouraging the hidden units to be independent and binary, eg Barlow et al (1989) and Schmidhuber (1992).”
Could information theory provide an ecological theory of sensory processing? (2011, background): “[7] Elegant examples of factorial codes can be found in Barlow et al. (1989) and Hentschel and Barlow (1991), see also Watanabe (1981, 1985).”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “Traditional unsupervised learning is about finding regular ities, by clustering the data, or encoding it through a factorial code [ 2 , 55] with statisti cally independent components, or predicting parts of it from other parts.”
Simplicity: a unifying principle in cognitive science? (2003, background): “much of cognition concerns compression[23], or the elimination of redundancy[ 24 ], and”
Binding via Reconstruction Clustering (2015, background): “Complementary to that, disentangling (Barlow et al., 1989; Schmidhuber, 1992; Bengio et al., 2007) requires the factors of variation in the data to be separated into different independent features.”