Normalization as a canonical neural computation

By Matteo Carandini, David J. Heeger. 2011

1,924 citations recorded by the source corpus.

There is increasing evidence that the brain relies on a set of canonical neural computations, repeating them across brain regions and modalities to apply similar operations to different problems. A promising candidate for such a computation is normalization, in which the responses of neurons are divided by a common factor that typically includes the summed activity of a pool of neurons. Normalization was developed to explain responses in the primary visual cortex and is now thought to operate throughout the visual system, and in many other sensory modalities and brain regions. Normalization may underlie operations such as the representation of odours, the modulatory effects of visual attention, the encoding of value and the integration of multisensory information. Its presence in such a diversity of neural systems in multiple species, from invertebrates to mammals, suggests that it serves as a canonical neural computation.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Optimization & Training

Citations in the corpus

Group Normalization (2018, background): “In fact, a well-accepted computational model in neuroscience is to normalize across the cell responses [21, 52, 55, 5], “with various receptive-field centers (covering the visual field) and with various spatiotemporal frequency tunings” (p183, [21]); this can happen not only in the primary visual co”
End-to-end Optimized Image Compression (2016, background): “These may be signiﬁcantly reduced through the use of joint local nonlinear gain control operations (Schwartz and Simoncelli, 2001; Lyu, 2010; Sinz and Bethge, 2013), inspired by models of visual neurons (Heeger, 1992; Carandini and Heeger, 2012).”
Density Modeling of Images using a Generalized Normalization Transformation (2015, background): “Speciﬁcally: • Choosing ε i ≡ 1 , α ij ≡ 1 , and γ ij ≡ 1 yields the classic form of the divisive normalization transformation (Carandini & Heeger, 2012), with exponents set to 1.”
Generalisation in humans and deep neural networks (2018, background): “These normalisation processes seem to be crucial for many aspects of robust animal and human vision [46], are predictive for human vision data [21, 64] and have proven useful in the context of computer vision [65, 66].”
How does the brain solve visual object recognition? (2012, neutral): “…level neuronal response patterns can be achieved with extensions to the simple LN model framework—most notably, by divisive normalization schemes in which the output of each LN neuron is normalized (e.g., divided) by aweighted sumof apool of nearby neurons (reviewed by Carandini and Heeger, 2011).”
Batch-normalized Maxout Network in Network (2015, background): “Normalization also exists in biological neural network, which is a canonical neural computation well-studied in neuroscience ﬁeld [1].”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, methodology): “A and B in standard divisive normalization are referred to as summation and suppression ﬁelds (Carandini & Heeger, 2012).”
Deep convolutional models improve predictions of macaque V1 responses to natural images (2017, background): “However, previous work investigating these normalization pools has 348 employed simple stimuli such as gratings [18] and we are not aware of any work learning 349 the entire normalization function from neural responses to natural stimuli.”
The atoms of neural computation (2014, background): “Divisive normalization Shunting inhibition in networks, balanced background synaptic excitation and inhibition or other mechanisms (Carandini & Heeger, 2012) Common across many cortical areas Sequencing of events over time Feed-forward cascades Synfire chains (Ikegaya et al., 2004) Language and…”
Computing with a Canonical Neural Circuits Model with Pool Normalization and Modulating Feedback (2014, background): “These observations lead to the recent proposal (Carandini & Heeger, 2012) that such normalization mechanisms define a canonical principle in cortical computation captured by the steady-state mechanism r i = β · g ex ( k s k · + ik ) + act bias α + g in ( k y k · − ik ) , (2.2) where s and y denote…”
End-to-End Blind Image Quality Assessment Using Deep Neural Networks (2018, methodology): “These higher-order statistics may be significantly decorrelated through the use of a joint nonlinear gain control mechanism [45], [46] inspired by models of visual neurons [47], [48].”
Recurrent Processing during Object Recognition (2013, methodology): “Thus,we ﬁnd evidence for the importance of recurrent inhibitory competition in providing dynamic renor-malization of network response over a wide range of input signal strengths (Carandini and Heeger, 2012).”
The Forward-Forward Algorithm: Some Preliminary Investigations (2022, methodology): “To prevent this, FF normalizes the length of the hidden vector before using it as input to the next layer (Ba et al., 2016b; Carandini and Heeger, 2013) This removes all of the information that was used to determine the goodness in the ﬁrst hidden layer and forces the next hidden layer to use…”
A mechanistic cortical microcircuit of attention for amplification, normalization and suppression. (2015, methodology): “The neural responses in the model largely follow concepts of divisive normalization (Carandini & Heeger, 2012; Reynolds & Heeger, 2009).”
A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives. (2015, background): “This simplification makes the assumption that responses to each region of the visual field are independent, which is not the case at the level of single cells (Carandini and Heeger, 2011; Albright and Stoner, 2002).”