Sparse coding with an overcomplete basis set: a strategy employed by V1?

By Barbara Olshausen, David J. Field. 1997

The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and bandpass, comparable with the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images. Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete--i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input-output function will deviate from being purely linear. These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.

Subfields: Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neuroscience

Citations in the corpus

Remote Sensing Image Scene Classification: Benchmark and State of the Art (2017, methodology): “but not limited to, principal component analysis (PCA) [135], k-means clustering, sparse coding [136], and autoencoder [137].”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (2016, background): “The direct mapping approach focuses on the latter aspect and is typically built via autoencoders [6,17,25].”
A Theory of Generative ConvNet (2016, methodology): “In fact, a main motivation for this paper is to reconcile the FRAME model (Zhu et al., 1997), where the Gabor wavelets play the role of bottom-up filters, and the Olshausen-Field model (Olshausen & Field, 1997), where the wavelets play the role of top-down basis functions.”
Pedestrian Detection with Unsupervised Multi-stage Feature Learning (2012, background): “Following [30] many authors have used sparse dictionary learning to represent images [27, 1, 19].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “…to train the ﬁrst few layers of a deep neural networks using a single training image , such as this Image A , B or even C (above), provided that sufﬁcient data augmentation is used. learned and hand-crafted features for visual recognition (Olshausen & Field, 1997; Lowe, 2004; Dalal & Triggs, 2005).”
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (2015, methodology): “For image processing tasks, models based on marginal distributions of image gradients are effective [20, 25], but are only designed for image restoration rather than being true density models (so cannot sample an actual image).”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, methodology): “We use a data model called sparse coding, which is a popular model to model image, text and speech data [11, 12, 69, 77, 78, 100, 105, 106].”
Representation Learning: A Review and New Perspectives (2012, background): “ization permits signiﬁcant variance (above the nominal variance given by 1) only in the select directions of the sparsely activated h i. This is a property the ssRBM shares with sparse coding models (Olshausen and Field, 1997; Grosse et al., 2007) where the sparse latent representation also encodes ”
Learning to See by Moving (2015, background): “This hypothesis inspired research in fully unsupervised learning from visual imagery and led to popular algorithms such as ICA [5, 20], autoencoders [24, 9], sparse coding [32], restricted boltzmann machines [27, 37] and others [6].”
Learning to Generate Reviews and Discovering Sentiment (2017, background): “There is also a long history of unsupervised representation learning (Olshausen & Field, 1997).”
Video (language) modeling: a baseline for generative models of natural videos (2014, methodology): “The van Hateren dataset of natural videos has been a standard dataset for investigating models of temporal sequences (Cadieu & Olshausen, 2009; Olshausen & Field, 1997).”
Unsupervised Learning of Visual Representations Using Videos (2015, methodology): “Olhausen and Field [32] showed that using this criteria they can learn V1-like filters directly from the data.”
Early Methods for Detecting Adversarial Images (2016, background): “oling images are clearly distinct from typical whitened images, as is clear from Figure 2. The eect is similar in for YUV images. Note that some suggest that the human eye also whitens visual inputs [9], and neural networks may approximately whiten images too. Consequently, the whitened image may be”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “is believed to have evolved to discover efficient coding strategies [62].”