Large-Scale Feature Learning With Spike-and-Slab Sparse Coding

By Ian J. Goodfellow, Aaron C. Courville, Yoshua Bengio. 2012

96 citations recorded by the source corpus.

We consider the problem of object recognition with a large number of classes. In order to overcome the low amount of labeled examples available in this setting, we introduce a new feature learning and extraction procedure based on a factor model we call spike-and-slab sparse coding (S3C). Prior work on S3C has not prioritized the ability to exploit parallel architectures and scale S3C to the enormous problem sizes needed for object recognition. We present a novel inference procedure for appropriate for use with GPUs which allows us to dramatically increase both the training set size and the amount of latent factors that S3C may be trained with. We demonstrate that this approach improves upon the supervised learning capabilities of both sparse coding and the spike-and-slab Restricted Boltzmann Machine (ssRBM) on the CIFAR-10 dataset. We use the CIFAR-100 dataset to demonstrate that our method scales to large numbers of classes better than previous methods. Finally, we use our method to win the NIPS 2011 Workshop on Challenges In Learning Hierarchical Models' Transfer Learning Challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Challenges in representation learning: A report on three machine learning contests (2013, background): “While past workshop-based contests have had a low participation rate (example: 4 teams in the NIPS 2011 transfer learning challenge [20, 21]) the participation rate problem can be completely solved by hosting the contest on Kaggle.”
Semi-supervised Learning with Ladder Networks (2015, neutral): “31 Spike-and-Slab Sparse Coding (Goodfellow et al., 2012) 31.”
Learning with Pseudo-Ensembles (2014, methodology): “The winner of this challenge used convolutional Spike and Slab Sparse Coding, followed by max pooling and a linear SVM on the pooled features [6].”
Deep learning in neural networks: An overview (2014, background): “Exceptions include two 2011 contests (Goodfellow et al., 2011; Mesnil et al., 2011; Goodfellow et al., 2012) specialised on Transfer Learning from one dataset to another (e.”
Deep Learning of Representations: Looking Forward (2013, methodology): “Note also that sparse coding can be generalized into the spike-and-slab sparse coding algorithm (Goodfellow et al., 2012), in which MAP inference is replaced by variational inference, and that was used to win the NIPS 2011 transfer learning challenge (Goodfellow et al., 2011).”
My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013 (2013, methodology): “…are adaptations and extensions of an over 40-year-old algorithm, namely, supervised efﬁ-cient backprop [59, 90] (TL 1.2) (compare [60, 53, 75, 12, 67]) or BPTT/RTRL for RNN, e.g., [94, 73, 91, 68, 80, 61] (exceptions include two 2011 contests specialised on transfer learning [35]—but compare [23]).”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, background): “In addition, in the two transfer learning challenges held in 2011, learning algorithms leveraging deep architectures exhibited superior performance compared to existing learning algorithms [54], [55].”
Deep Growing Learning (2017, unknown): “, Spike-and-Slab Sparse Coding [8] and View-Invariant k-means [16].”
Deep Feature Selection: Theory and Application to Identify Enhancers and Promoters (2015, unknown): “Other sparse regularization techniques, for example, spike-and-slab (Goodfellow et al., 2012), can be tested to achieve a better quality of sparseness.”
Beyond Manual Tuning of Hyperparameters (2015, background): “, [9, 22, 43, 46, 56]); likewise, network architectures and their interactions may be regarded as a result of a probabilistic inversion of hierarchical generative models for structured data (compare, e.”
New Algorithms for Learning Incoherent and Overcomplete Dictionaries (2013, unknown): “…The requirement that nonzero X i ’s be bounded away from zero in magnitude is similar in spirit to the Spike-and-Slab Sparse Coding (S3C) model of [27], which also encourages nonzero latent variables to be bounded away from zero to avoid degeneracy issues that arise when some coeﬃcients are much…”
Encoding Tree Sparsity in Multi-Task Learning: A Probabilistic Framework (2014, methodology): “We are also interested in the comparison between the proposed model and some other sparse feature learning methods, e.g. the sparse coding method (Goodfellow, Courville, and Bengio 2012).”
Recurrent convolutional neural network for object recognition (2015, background): “There is an interesting relationship between RCNN and some sparse coding models [15] where fixed-point updates are used in inference.”