Sparse Coding via Thresholding and Local Competition in Neural Circuits

By Christopher J. Rozell, Don H. Johnson, Richard Baraniuk, Barbara Olshausen. Neural Computation, 2008

515 citations recorded by the source corpus.

Subfields: Cognitive Neuroscience, Computational Mechanics, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/20b003b91679fd88b63e9bfab6f05f1e7ecabe2c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computational Mechanics

In Electrical and Electronic Engineering

Citations in the corpus

Statistical mechanics of complex neural systems and high dimensional data (2013, methodology): “these same techniques to neuronal information processing are discussed in [30], including semantic information processing [42, 43, 44], short-term memory [45, 46], neural circuits for L1 minimization [47], learning sparse representations [48, 49], regularized learning of high dimensional synaptic we”
Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. (2012, background): “Here we describe one plausible two-layer circuit solution (see Figure 5a) proposed in Rozell et al. (2008), inspired by gradient descent in s on the cost function in Equation 3.”
The Visual System's Internal Model of the World (2015, background): “O’ Reilly et al.’s Leabra cognitive architecture [17] may be the most powerful instantiation of this class of models for recurrent processing in object recognition.”
Neural Expectation Maximization (2017, methodology): “Gradient backpropagation through inference updates has previously been addressed in the context of sparse coding with (Fast) Iterative Shrinkage/Tresholding Algorithms ((F)ISTA; [5, 27, 2]).”
Training Deep Spiking Neural Networks Using Backpropagation (2016, background): “Thus, as soon as any neuron produces an output spike, it inhibits all other neurons in the circuit and prevents them from spiking [25].”
Understanding Deep Architectures using a Recursive Convolutional Network (2013, background): “[20] showed how the ISTA scheme can be unwrapped into a repeated series of network layers, which can be viewed as a recursive net.”
Sparseness and expansion in sensory representations. (2014, background): “Our Hebbian model for J plus the subsequent threshold nonlinearity can be viewed as a simple, biologically plausible approximation to the more complex machine learning algorithms for sparse reconstruction (Ganguli and Sompolinsky, 2012; Rozell et al., 2008).”
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, background): “One simple attractor network that performs the useful nontrivial computation in its dynamics is the LCA [52].”
Rectified Factor Networks (2015, methodology): “The second problem is that sparse priors like Laplacians for the coding units are computationally very expensive when calculating the code for a new input (Olshausen & Field, 1996; Hoyer, 2004; Rozell et al., 2008; Gregor & LeCun, 2010).”
Deeply Improved Sparse Coding for Image Super-Resolution (2015, methodology): “…according to where h θ is an element-wise shrinkage function deﬁned as Different from the iterative shrinkage and thresholding algorithm (ISTA) [6, 23] which ﬁnds an analytical relationship between network parameters (weights W , S and thresholds θ ) and sparse coding parameters ( D y and λ ),…”
On rectified linear units for speech processing (2013, methodology): “This method can be interpreted as a special case of PSD [12, 13] and sparse coding [14] when inference of features is computed in just one step.”
Learning Fast Approximations of Sparse Coding (2010, methodology): “, 2004; Beck & Teboulle, 2009), and (Rozell et al., 2008) for a continuous-time, biologically relevant form of ISTA).”