The Sparseness of Mixed Selectivity Neurons Controls the Generalization-Discrimination Trade-Off

By Omri Barak, Mattia Rigotti, Stefano Fusi. 2013

230 citations recorded by the source corpus.

Intelligent behavior requires integrating several sources of information in a meaningful fashion—be it context with stimulus or shape with color and size. This requires the underlying neural mechanism to respond in a different manner to similar inputs (discrimination), while maintaining a consistent response for noisy variations of the same input (generalization). We show that neurons that mix information sources via random connectivity can form an easy to read representation of input combinations. Using analytical and numerical tools, we show that the coding level or sparseness of these neurons' activity controls a trade-off between generalization and discrimination, with the optimal level depending on the task at hand. In all realistic situations that we analyzed, the optimal fraction of inputs to which a neuron responds is close to 0.1. Finally, we predict a relation between a measurable property of the neural representation and task performance.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “By contrast, high-dimensional neural codes may emerge by preference in tasks with minimal need for generalization, such as recall and recognition of a small set of unrelated images (Barak et al., 2013; Rigotti et al., 2013).”
Re-evaluating circuit mechanisms underlying pattern separation (2019, neutral): “Thresholding nonlinearities can increase the dimensionality of the coding subspace by decorrelating activity, but excessive sparsening can ultimately limit the extent of the coding subspace (Barak et al., 2013; Babadi et al., 2014; Billings et al., 2014; Cayco-Gajic et al., 2017; Litwin-Kumar et al.”
Sparseness and expansion in sensory representations. (2014, unknown): “A finite optimal sparseness has been also found numerically in a model for processing inputs that lie in low dimensional subspace by projecting them with random weights (Barak et al., 2013).”
The geometry of abstraction in hippocampus and prefrontal cortex (2020, background): “This is because, at the level of individual neurons, the linear component of responses dominates in many situations (Rigotti et al., 2010, 2013; Barak et al., 2013; Lindsay et al., 2017; Fusi et al., 2016), and it is easy to miss the fact that non-linear com-”
Neural population coding: combining insights from microscopic and mass signals (2015, background): “…trade-off between the need to maximize the diversity of responses to different stimuli (increasing the stimulus discrimination properties of population responses), and the need to maintain high response reliability (i.e., achieving consistent responses for noisy variations of the input) [37].”
Computational principles of biological memory (2015, background): “is is no longer true when we consider neural circuits of a signiﬁcantly larger size N n. Moreover, sparseness can also be beneﬁcial in terms of generalization, but only if fis not too small (see e.g. Barak et al., 2013). For these reasons, sparse representations are unlikely to be the sole solution ”
A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses (2015, background): “Clearly, the generalization properties of our system are not comparable to standard machine learning approaches but they are also expected to scale with the number of randomly connected neurons in the hidden layer (Rigotti et al., 2010; Barak et al., 2013).”
Computational principles of synaptic memory consolidation (2016, background): “It is not unreasonable to think that these mechanisms can also greatly contribute to the decorrelation of synaptic modifications.”
Cortical feedback decorrelates olfactory bulb output in awake mice (2015, background): “Within this framework, the piriform cortex may act as a pattern-recognition device (Babadi and Sompolinsky, 2014; Barak et al., 2013; Caron et al., 2013; Haberly, 2001; Haberly and Bower, 1989; Linster and Hasselmo, 2001) that compares incoming sensory inputs with representations of previously…”
Computational models of long term plasticity and memory (2017, background): “Moreover, sparseness can also be beneficial in terms of generalization (see e.g.[Olshausen and Field, 2004]), but only if f is not too small [Barak et al., 2013].”
Local Receptive Fields Based Extreme Learning Machine (2015, background): “Solid biological evidence also justifies the local receptive fields, which shows that each cell in the visual cortex is only sensitive to a corresponding sub-region of the retina (input layer) [54]–[56].”