Semilinear Predictability Minimization Produces Well-Known Feature Detectors

By J. Urgen Schmidhuber, Martin Eldracher, Bernhard Foltin. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “Such codes may be sparse and can be advantageous for (1) data compression, (2) speeding up subsequent BP (Becker, 1991), (3) trivialising the task of subsequent naive yet optimal Bayes classifiers (Schmidhuber et al., 1996).”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentanglement is mostly about how information is encoded in the representation, and it is often argued that a representation that is disentangled is desirable in learning to solve challenging real-world down-stream tasks [6, 73, 59, 7, 26, 68].”
High-Performance Neural Networks for Visual Object Classification (2011, methodology): “…best results with a contrast-extraction layer (Fukushima, 2003) realized by Mexican hat-shaped ﬁlters of size 21 × 21, one with a concentric on-center receptive ﬁeld and one with a concentric oﬀ-center receptive ﬁeld, Schmid-huber, 1992) applied to natural images (Schmidhuber et al. , 1996).”
Censoring Representations with an Adversary (2015, methodology): “The experiments in this work were on synthetic data, and were later followed up to learn ﬁlters from natural image patches in Schmidhuber et al. (1996) and Schraudolph et al. (1999).”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, background): “, 2018; Schwartz & Simoncelli, 2001) for recent efforts), and has led to a number of algorithms for supervised and unsupervised learning (Ballé et al., 2017; Deco & Parra, 1997; Földiák, 1990; Linsker, 1988; Redlich, 1993a;b; Schmidhuber et al., 1996).”
Self-Delimiting Neural Networks (2012, background): “…Neocognitron [13]; they sometimes (but not always) proﬁt from sparse network connectivity and techniques such as weight sharing & convolution [38, 1], max-pooling [49], and contrast enhancement like the one automatically generated by unsupervised Predictability Minimization [53, 64, 67].”
A committee of neural networks for traffic sign classification (2011, methodology): “Unsupervised learning methods applied to patches of natu-ral images tend to produce localized ﬁlters that resemble off-center-on-surround ﬁlters, orientation-sensitive bar detectors, Gabor ﬁlters [2], [3], [4].”
The "independent components" of natural scenes are edge filters. (1997, unknown): “Another principle, predictability minimization, has also been brought to bear on the problem by Schmidhuber et al. (1996).”
Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction (2011, methodology): “Methods using this paradigm include stacks of: Low-Complexity Coding and Decoding machines (LOCOCODE) [10], Predictability Minimization layers [23,24], Restricted Boltzmann Machines (RBMs) [8], auto-encoders [20] and energy based models [15].”
Multi-column deep neural network for traffic sign classification (2012, methodology): “Unsupervised learning methods applied to patches of natural images tend to produce localized filters that resemble off-center-on-surround filters, orientation-sensitive bar detectors, Gabor filters (Hoyer & Hyvärinen, 2000; Olshausen & Field, 1997; Schmidhuber, Eldracher, & Foltin, 1996).”
Feature Extraction Through LOCOCODE (1999, background): “LETTER Communicated by Richard Zemel”
Nonlinear ICA through low-complexity autoencoders (1999, background): “For the authors’ alternative neural approaches to nonlinear ICA see [29, 30].”
On Fast Deep Nets for AGI Vision (2011, background): “We sometimes (but not always) proﬁt from sparse network connectivity and techniques such as weight sharing & convolution [12, 1, 25], max-pooling [17], and contrast enhancement [6] like the one automatically generated by unsupervised Predictability Minimization [18, 22, 24].”