Self-organization in a perceptual network

By Ralph Linsker. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neuroscience

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, methodology): “Furthermore, it can be shown that I ( g 1 ( X (1) ); g 2 ( X (2) )) ≤ I ( X ; g 1 ( X (1) ) , g 2 ( X (2) )) , 2 hence the objective in (2) can be seen as a lower bound on the InfoMax objective max g ∈G I ( X ; g ( X )) (Linsker, 1988).”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “The infomax optimization principle (Linsker, 1988; Bell & Sejnowski, 1995), as prescribed for neural networks, advocates maximizing MI between the input and output.”
What makes for good views for contrastive learning (2020, background): “This principle also complements the already popular “InfoMax principle" [34] , which states that a goal in representation learning is to capture as much information as possible about the stimulus.”
Deep learning in neural networks: An overview (2014, neutral): “Many UL methods are designed to maximize entropy-related, information-theoretic (Boltzmann, 1909; Kullback & Leibler, 1951; Shannon, 1948) objectives (e.g., Amari, Cichocki, & Yang, 1996; Barlowet al., 1989;Dayan&Zemel, 1995;Deco&Parra, 1997; Field, 1994; Hinton, Dayan, Frey, & Neal, 1995; Linsker, ”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, methodology): “This concept is encapsulated by the InfoMax objective [16]:”
Learning Factorial Codes by Predictability Minimization (1992, methodology): “This has already been noted by Linsker (1988) in the context of his Infomax principle.”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, background): “, 2018; Schwartz & Simoncelli, 2001) for recent efforts), and has led to a number of algorithms for supervised and unsupervised learning (Ballé et al., 2017; Deco & Parra, 1997; Földiák, 1990; Linsker, 1988; Redlich, 1993a;b; Schmidhuber et al., 1996).”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “If there is no speciﬁc meaning to embed in the desired representation, the infomax principle , described in Linsker [1988], states that an optimal representation is one of bounded entropy which retains as much information about the data as possible.”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, methodology): “Such a framework is inspired by the InfoMax principle (Linsker, 1988) and has been the main driver of progress in self-supervised representation learning in other domains such as computer vision, audio processing, and reinforcement learning (Belghazi et al.”
Formal Limitations on the Measurement of Mutual Information (2018, background): “ic coding function Cx so as to maximize I(Cx(x),y) while minimizing I(Cx(x),x). Here one does not ask for a coding function on y and one does not limit H(Cx(x)). Another related framework is INFOMAX (Linsker, 1988; Bell & Sejnowski, 1995; Hjelm et al., 2018). Here we consider a population distri”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, methodology): “Conventional wisdom says that representations should extract the most shared information between positive pairs and remain invariant to other noise factors (Linsker, 1988; Tian et al., 2019; Wu et al., 2020; Bachman et al., 2019).”
Maximum Entropy Generators for Energy-Based Models (2019, background): “With x = G ( z ) the function of interest and x discrete-valued: This idea of maximizing mutual information between inputs and outputs of an encoder to maximize the entropy of its output dates back at least to Linsker (1988); Bell & Sejnowski (1995) and their work on Infomax.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “InfoNCE (1) has an interpretation as multi-class logistic regression, and lower bounds the mutual information across similar views (z̃, z̃0)—a common representation learning objective [4, 9, 15, 50, 75, 79, 80, 97, 120].”
Finding a roadmap to achieve large neuromorphic hardware systems (2013, methodology): “…area treating neurons as an ANN type model with different learning rules ﬁnding patterns corresponding to Principle Component Analysis (PCA) (e.g., Linsker, 1988; MacKay and Miller, 1990), Independent Component Analysis (ICA) (e.g., Bell and Sejnowski, 1995, 1997; Hoyer and Hyvarinen, 2000), and…”
An Optimality Principle for Unsupervised Learning (1988, methodology): “The optimality principle which we will use to describe the algorithm is based on the idea of maximizing information which was first proposed as a desirable property of neural networks by Linsker (1986, 1988).”
The "independent components" of natural scenes are edge filters. (1997, unknown): “The first work along these lines was by Linsker (1988), who first proposed the "infomax" principle which underlies our own work.”
The `independent components''of natural scenes are edge filters (1997, unknown): “The rst work along these lines was by Linsker (1988) who rst proposed thèinfomax' principle which underlies our own work.”
Self-supervised Learning from a Multi-view Perspective (2020, methodology): “Our method also resembles the InfoMax principle (Linsker, 1988; Hjelm et al., 2018) and the Multi-view Information Bottleneck method (Federici et al., 2020).”