The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization

By Adam Coates, Andrew Y. Ng. 2011

639 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition (2014, background): “Index Terms—Convolutional Neural Networks, Spatial Pyramid Pooling, Image Classification, Object Detection F”
Improving neural networks by preventing co-adaptation of feature detectors (2012, neutral): “5% [4].”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “Computer vision, in contrast, further concerns dictionary building [54, 9, 5], as the raw signal is in a continuous, high-dimensional space and is not structured for human communication (e.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, methodology): “classes excluding cat and truck, which closely matches the SVM-based classification results in the original Coates and Ng paper [6] that used all 10 classes.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, neutral): “, “transductive” models [14, 22, 21], graph-based methods [49, 4, 29], generative modeling [3, 27, 41, 9, 17, 23, 38, 34, 42], etc.”
Representation Learning: A Review and New Perspectives (2012, methodology): “The mcRBM is a combination of the covariance RBM (cRBM) (Ranzato et al., 2010a), that models the conditional covariance, with the GRBM that captures the conditional mean.”
Multi-column deep neural networks for image classification (2012, methodology): “[8]; 5.60% - [32] No preprocessing is used for this dataset.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, positive): “A common practice in the Deep Learning literature (Ranzato et al., 2007, 2008b; Lee et al., 2008, 2009; Bagnell and Bradley, 2009; Glorot et al., 2011a; Coates and Ng, 2011; Goodfellow et al., 2011) consists in adding a penalty term to the training criterion that encourages the hidden units to be sp”
Training Deep Neural Networks on Noisy Labels with Bootstrapping (2014, methodology): “In all experiments we ﬁrst extracted spatial-pyramid-pooled OMP-1 features as described in (Coates & Ng, 2011) to get 3200 -dimensional features.”
Learning Invariant Representations with Local Transformations (2012, background): “5% deep NN (Coates & Ng, 2011b) 82 .”
Deep Learning of Representations: Looking Forward (2013, background): “It is interesting to note the results obtained by Coates and Ng (2011) which suggest that sparse coding is a better encoder but not a better learning algorithm than RBMs and sparse auto-encoders (none of which has explaining away).”
Understanding Deep Architectures using a Recursive Convolutional Network (2013, background): “All layers use rectified linear units (ReLU) [3, 4, 16].”
Part Detector Discovery in Deep Convolutional Neural Networks (2014, neutral): “Non-linear layers, such as the rectiﬁed linear unit used by [1], are related to non-linear encoding techniques for BoW [24].”
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding (2012, methodology): “We compare S3C to two other feature extraction methods: OMP-1 with thresholding, which Coates and Ng (2011) found to be the best feature extractor on CIFAR-10, and sparse coding, which is known to perform well when less labeled data is available.”
Unsupervised feature learning framework for no-reference image quality assessment (2012, methodology): “The soft-assignment function used here can be considered as a special case of the soft threshold function in [5] where we set the adjustable threshold to zero.”
Learning to Disentangle Factors of Variation with Manifold Interaction (2014, methodology): “For experiments on TFD, we built a 2-layer model whose first layer is constructed with convolutional features extracted using the filters trained with OMP-1 followed by 4×4 max pooling (Coates & Ng, 2011).”
Multimodal Neural Language Models (2014, neutral): “We follow the pipeline of [22].”
k-Sparse Autoencoders (2013, methodology): “However, it has been observed that often, better results are obtained when the sparse encoding stage used for classiﬁcation does not exactly match the encoding used for dictionary training (Coates & Ng, 2011).”
Super Normal Vector for Activity Recognition Using Depth Sequences (2014, methodology): “We choose sparse coding over GMM in our aggregation scheme because it is cheaper to compute the centers (dictionary), especially it was recently shown in [2] that a reasonably good dictionary can be created by some simple methods, e.”