Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

By Ian J. Goodfellow, Aaron C. Courville, Yoshua Bengio. 2012

61 citations recorded by the source corpus.

We consider the problem of using a factor model we call {\em spike-and-slab sparse coding} (S3C) to learn features for a classification task. The S3C model resembles both the spike-and-slab RBM and sparse coding. Since exact inference in this model is intractable, we derive a structured variational inference procedure and employ a variational EM training algorithm. Prior work on approximate inference for this model has not prioritized the ability to exploit parallel architectures and scale to enormous problem sizes. We present an inference procedure appropriate for use with GPUs which allows us to dramatically increase both the training set size and the amount of latent factors. We demonstrate that this approach improves upon the supervised learning capabilities of both sparse coding and the ssRBM on the CIFAR-10 dataset. We evaluate our approach's potential for semi-supervised learning on subsets of CIFAR-10. We demonstrate state-of-the art self-taught learning performance on the STL-10 dataset and use our method to win the NIPS 2011 Workshop on Challenges In Learning Hierarchical Models' Transfer Learning Challenge.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, neutral): “, “transductive” models [14, 22, 21], graph-based methods [49, 4, 29], generative modeling [3, 27, 41, 9, 17, 23, 38, 34, 42], etc.”
Representation Learning: A Review and New Perspectives (2012, methodology): “We would like to use the data, along with very generic priors, to discover those features, or equivalently, a similarity function.”
Deep learning in neural networks: An overview (2014, background): “Exceptions include two 2011 contests specialised on Transfer Learning (Goodfellow et al., 2011; Mesnil et al., 2011; Goodfellow et al., 2012)—but compare SL-based transfer in deep NN (Ciresan et al.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “A common practice in the Deep Learning literature (Ranzato et al., 2007, 2008b; Lee et al., 2008, 2009; Bagnell and Bradley, 2009; Glorot et al., 2011a; Coates and Ng, 2011; Goodfellow et al., 2011) consists in adding a penalty term to the training criterion that encourages the hidden units to be sp”
Joint Unsupervised Learning of Deep Representations and Image Clusters (2016, methodology): “This performance also beats several other methods listed in [4, 15, 25, 34].”
Deep Learning of Representations: Looking Forward (2013, methodology): “Note also that sparse coding can be generalized into the spike-and-slab sparse coding algorithm (Goodfellow et al., 2012), in which MAP inference is replaced by variational inference, and that was used to win the NIPS 2011 transfer learning challenge (Goodfellow et al., 2011).”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, methodology): “The S3C model possesses a set of latent binary spike variables h 2 f 0 ; 1 g d h , a set of latent real-valued slab variables s 2 R d h , and real-valued d x -dimensional visible vector x 2 R d x .”
Learning Feature Representations with K-Means (2012, methodology): “Speciﬁcally, when we use the single-layer architecture outlined above, it turns out that algorithms like sparse coding and more sophisticated variants (e.g., spike-slab sparse coding [20]) are diﬃcult to top when we have very little labeled data.”
Multimodal Transitions for Generative Stochastic Networks (2013, background): “This advantage clearly shows up (e.g., as illustrated in the transfer learning competitions with unsupervised deep learning [6, 25, 18]) in practical applications where the distribution changes or new classes or domains are considered (transfer learning, domain adaptation), when some classes are…”
Realistic Evaluation of Semi-Supervised Learning Algorithms (2018, methodology): “…and algorithms based on learning features (frequently via generative modeling) from unlabeled data (Belkin & Niyogi, 2002; Lasserre et al., 2006; Salakhutdinov & Hinton, 2007; Coates & Ng, 2011; Goodfellow et al., 2011; Kingma et al., 2014; Pu et al., 2016; Odena, 2016; Salimans et al., 2016).”