Emergence of simple-cell receptive field properties by learning a sparse code for natural images

By Barbara Olshausen, David J. Field. 1996

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “Secondly, neural networks have been successfully trained under a wide range of adversarial regularisation constraints, including sparse internal states [23], stochastically masked units [28] and randomly perturbed weights [1].”
What makes ImageNet good for transfer learning? (2016, background): “The most well-known such criteria are image reconstruction [7, 38, 31, 29, 34, 23] (see [6] for a comprehensive overview) and feature slowness [48, 18].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “The intuition that a good representation should faithfully reconstruct the input data was soon exploited to achieve representations that resemble simple cells in the brain [48].”
Unsupervised Visual Representation Learning by Context Prediction (2015, methodology): “Sparse autoencoders also use reconstruction as a pretext task, along with a sparsity penalty [39], and such autoencoders may be stacked to form a deep representation [32, 31].”
Generative Visual Manipulation on the Natural Image Manifold (2016, methodology): “A common strategy is to learn local filter or patch models, such as Principal Components, Independent Components, Mixture of Gaussians or wavelet bases [14,15,16].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “, using sparse coding [58], adversarial training [12, 13, 50], autoencoders [49, 63, 76], or probabilistic versions thereof [67].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “This can be modeled as maximizing the probability of reconstructing the input [43, 51, 67] and optionally estimating latent variables [29, 58] or using adversarial training [15, 44].”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “In contrast, dictionary learning techniques, such as sparse [37] and non-negative [36] decompositions, express each input pattern using a combination of a small number of selected features out”
Representation Learning: A Review and New Perspectives (2012, methodology): “Indeed, as shown by one of us (Vincent, 2011) and others (Swersky et al., 2011), in the case of the restricted Boltzmann machine, training the model via an inductive principle known as score matching (Hyvärinen, 2005) (to be discussed in sec.”
Building high-level features using large scale unsupervised learning (2011, methodology): “One shortcoming of early approaches such as sparse coding (Olshausen & Field, 1996) is that their architectures are shallow and typically capture low-level concepts (e.g., edge “Gabor” filters) and simple invariances.”
Learning to See by Moving (2015, background): “This hypothesis inspired research in fully unsupervised learning from visual imagery and led to popular algorithms such as ICA [5, 20], autoen-coders [24, 9], sparse coding [32], restricted boltzmann machines [27, 37] and others [6].”
Tutorial on Variational Autoencoders (2016, background): “sparse autoencoders [10, 11] or denoising autoencoders [12, 13].”
Deep learning in neural networks: An overview (2014, neutral): “…1997; Hochreiter & Schmidhuber, 1999; Hyvärinen, Hoyer, & Oja, 1999; Lewicki & Olshausen, 1998) throughwell-known feature detectors (e.g., Olshausen & Field, 1996; Schmidhuber, Eldracher, & Foltin, 1996), such as off-center-on-surround-like structures, aswell as orientation sensitive…”
Recent advances in convolutional neural networks (2015, background): “Famous variants of AE include sparse AE [81, 82], Denoising AE [78] and Variational AE (VAE) [73].”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “ing based on this behavior was first proposed in [63].”
View-tolerant face recognition and Hebbian learning imply mirror-symmetric neural tuning to head orientation (2016, neutral): “The class of learning rules yielding invariance includes those that emerge from principles such as sparsity and the efficient coding hypothesis [6, 7, 39].”
On the Binding Problem in Artificial Neural Networks (2020, background): “In this way learned object representations can also benefit from known inductive biases that focus on feature hierarchies, invariances, and spatio-temporal coherence (Becker and Hinton, 1992), sparsity (Olshausen and Field, 1996), or non-Euclidean feature spaces (Nickel and Kiela, 2017).”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”