An Application of the Principle of Maximum Information Preservation to Linear Systems

By Ralph Linsker. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

Auto-Encoding Variational Bayes (2013, positive): “In relevant recent work on autoencoders [VLL + 10] it was shown that the training criterion of un-regularized autoencoders corresponds to maximization of a lower bound (see the infomax principle [Lin89]) of the mutual information between input X and latent representation Z .”
Learning Robust and Multilingual Speech Representations (2020, neutral): “learn representations that describe data by maximizing mutual information between data and representation variables, have been explored for a long time in unsupervised representation learning (Linsker, 1988; Bell and Sejnowski, 1995).”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “We first describe the relationship between our framework and the celebrated infomax principle [18] applied to representation learning [28], stating that “good representation” is likely to contain much information in the corresponding input.”
Learning a Mahalanobis Metric from Equivalence Constraints (2005, methodology): “Information theoretic criteria for unsupervised learning in neural networks were studied by Linsker (1989), and have been used since in several tasks in the neural network literature.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “InfoNCE (1) has an interpretation as multi-class logistic regression, and lower bounds the mutual information across similar views (z̃, z̃0)—a common representation learning objective [4, 9, 15, 50, 75, 79, 80, 97, 120].”
Visual adaptation: neural, psychological and computational aspects. (2007, background): “…function of the optimal ﬁlter for a linear system is inversely proportional to the amplitude spectrum of the stimulus distribution up to some cut-oﬀ frequency beyond which neural noise dominates (Atick & Red-lich, 1990, 1992; Barlow, 1990; Linsker, 1989; Srinivasan, Laughlin, & Dubs, 1982).”
Redundancy Reduction as a Strategy for Unsupervised Learning (1993, methodology): “Some first steps in this direction have been taken by Linsker (1989) and by Atick and Redlich (1990a), both using the mutual information between the desired scene data and the noisy image signal (for a different application of redundancy reduction to supervised learning, see Redlich 1992).”
An Introduction to Variational Autoencoders (2019, background): “, 2010) it was shown that the training criterion of unregularized autoencoders corresponds to maximization of a lower bound (see the infomax principle (Linsker, 1989)) of the mutual information between input X and latent representation Z.”
Distance Metric Learning: A Comprehensive Survey (2006, methodology): “For a given testing sample x0, the task reduces to estimating the class posterior probabilitiesP (j|x0)Jj=1.”
Could information theory provide an ecological theory of sensory processing? (2011, background): “…in the nervous system potentially has evolutionary advantages (Attneave 1954; Barlow 1961, 1985; Uttley 1979; Srinivisan et ai 1982; Field 1987; Linsker 1988, 1989a, b; Atick and Redlich 1990a, b; 1992a; Atick et al. 1990, 1991; Bialek et al. 1991b, see also Barlow 1989 and references therein)…”
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion (2010, methodology): “This is the infomax principle put forward by Linsker (1989).”
Information Maximization in Noisy Channels : A Variational Approach (2003, background): “In a neural context, maximal information transmission has been extensively studied and proposed as a principal goal of sensory processing[3, 9, 7, 11, 12].”
Learning Distance Functions using Equivalence Relations (2003, methodology): “Information theoretic criteria for unsupervised learning in neural networks were first suggested by (Linsker, 1989), and has been used since in several tasks in the neural network literature, e.”
Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context (1997, background): “This work owes much in terms of inspiration to the work of W. Grey Walter [46,47,48], Valentino Braitenberg [3], Richard Dawkins [9,10,11], John Holland [21], Ralph Linsker [29,30,31], and John Pearson [37].”
Continuous latent variable models for dimensionality reduction and sequential data reconstruction (2001, background): “31), or approximate versions of it, can be justified from several points of view: Information maximisation In the low noise case, the infomax principle states that if the mutual information between the inputs t and the outputs y of a processor is maximum, then the output distribution p(y) is factori”