Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

By Koray Kavukcuoglu, Marc'Aurelio Ranzato, Yann LeCun. 2010

252 citations recorded by the source corpus.

Adaptive sparse coding methods learn a possibly overcomplete set of basis functions, such that natural image patches can be reconstructed by linearly combining a small subset of these bases. The applicability of these methods to visual object recognition tasks has been limited because of the prohibitive cost of the optimization algorithms required to compute the sparse representation. In this work we propose a simple and efficient algorithm to learn basis functions. After training, this model also provides a fast and smooth approximator to the optimal representation, achieving even better accuracy than exact sparse coding algorithms on visual object recognition tasks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Auto-Encoding Variational Bayes (2013, positive): “Related are also encoder-decoder architectures such as the predictive sparse decomposition (PSD) [KRL08], from which we drew some inspiration.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “…for efﬁcient generation of samples from the approximate posterior can also be seen as a probabilistic generalization of the approximate feed-forward inference methods developed for sparse coding models in the last few years (Kavukcuoglu et al., 2008; Bradley & Bagnell, 2008; Gregor & LeCun, 2010).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “This approach remained largely un-explored for nearly two decades, with some exceptions [34, 20].”
Pedestrian Detection with Unsupervised Multi-stage Feature Learning (2012, background): “Following [30] many authors have used sparse dictionary learning to represent images [27, 1, 19].”
Representation Learning: A Review and New Perspectives (2012, background): “The main idea can be summarized by the following equation for the training criterion, which is simultaneously optimized with respect to hidden codes (representation) h(t) and with respect to parameters (W,α): JPSD = ∑ t λ‖h(t)‖1+‖x(t)−Wh(t)‖22+‖h(t)− fα(x(t))‖22 (26) where x(t) is the input vector f”
Stacked What-Where Auto-encoders (2015, background): “This “what”/“where” concept also links to invariant feature hierarchies learning schemes [4, 5, 8, 9, 10, 21, 22].”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, background): “Due to these uncertainties, passing a single data sample through a learning system multiple times might lead to different predictions.”
Video (language) modeling: a baseline for generative models of natural videos (2014, methodology): “e to handle full resolution inputs, due to the curse of dimensionality. Other models attempt to overcome this problem by using priors over the features, such as sparsity (Olshausen & Field, 1997; Kavukcuoglu et al., 2008; Lee et al., 2007). Although these constraints make the learning problem be”
Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields (2010, methodology): “The training proceeds the same way as in the patch-based version of Kavukcuoglu et al. (2008).”
Deep Learning of Representations: Looking Forward (2013, background): “…present in the wake-sleep algorithm (Hinton et al., 1995; Frey et al., 1996; Hinton et al., 2006) in the context of variational inference for Sigmoidal Belief Networks and DBNs. Learned approximate inference is also crucial in the predictive sparse coding (PSD) algorithm (Kavukcuoglu et al., 2008).”
Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification (2016, background): “…were proposed, such as stacked (denoising) autoencoders (Bengio et al., 2007; Vin-cent et al., 2010), deep belief networks (Hinton et al., 2006; Lee et al., 2009), sparse encoder-decoders (Ranzato et al., 2007; Kavukcuoglu et al., 2010), and deep Boltzmann machines (Salakhutdinov & Hinton, 2009).”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “This is done by a deep nonlinear feature extractor trained in an unsupervised manner using the Predictive Sparse Decomposition [126] on intermediate datasets, where the amount of source data is gradually replaced by target samples.”
Distributed optimization of deeply nested systems (2012, background): “…Saad and Marom, 1990; Krogh et al., 1990; Rohwer, 1990) to recent work in learning sparse features (Olshausen and Field, 1996, 1997; Ranzato et al., 2007b; Kavukcuoglu et al., 2008) and dimensionality reduction (Carreira-Perpiñán and Lu, 2008, 2010, 2011; Wang and Carreira-Perpiñán, 2012).”
Mutual exclusivity loss for semi-supervised deep learning (2016, background): “One example is Predictive Sparse Decomposition (PSD) [11] for learning the filter coefficients in the filter bank layer.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “The most common unsupervised schemes used contrastive divergence (Hinton, 2002) methods (see Lee, Grosse, Ranganath, & Ng, 2009), sparse constraints (Ranzato et al., 2006, 2007), or PSD (Kavukcuoglu et al., 2010; LeCun et al., 2010).”
Learning Invariant Feature Hierarchies (2012, methodology): “We will concentrate on the concept of sparse auto-encoder, as implemented in the Predictive Sparse Decomposition (PSD) method [37,38,39].”
Image denoising with multi-layer perceptrons, part 2: training trade-offs and analysis of their mechanisms (2012, methodology): “Successful architectures for object recognition (Jarrett et al., 2009) also make use of sparse representations, in this case using a procedure called predictive sparse coding proposed by Kavukcuoglu et al. (2008).”
Large-scale deep unsupervised learning using graphics processors (2009, methodology): “…of recent work on scaling up DBN and sparse coding algorithms, sometimes with entire research papers devoted to ingenious methods devised speci.cally for each of these models (Hinton et al., 2006; Bengio et al., 2006; Murray & Kreutz-Delgado, 2006; Lee et al., 2006; Kavukcuoglu et al., 2008).”