ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning

By Quoc V. Le, Alexandre Karpenko, Jiquan Ngiam, Andrew Y. Ng. 2011

385 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

How transferable are features in deep neural networks? (2014, background): “This phenomenon occurs not only for different datasets, but even with very different training objectives, including supervised image classiﬁcation (Krizhevsky et al. , 2012), unsupervised density learning (Lee et al. , 2009), and unsupervised learning of sparse representations (Le et al. , 2011).”
Network In Network (2013, methodology): “In conventional CNN, this might be compensated by utilizing an over-complete set of filters [6] to cover all variations of the latent concepts.”
Predicting Parameters in Deep Learning (2013, methodology): “One of the models is ordinary RICA with no parameter prediction and the other has 50% of the parameters in each feature predicted using squared exponential kernel dictionary with a length scale of 1.0; since 50% of the parameters in each feature are predicted, the second model has twice as many feat”
Building high-level features using large scale unsupervised learning (2011, methodology): “In the simplest implementation, before processing each mini-batch a 4In (Bengio et al., 2007; Le et al., 2011a), the encoding weights and the decoding weights are tied: W1 = W2.”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “In our implementation of the CCA-RICA method (section 5) we used the provided codes by the authors of [4] for CCA and [28] for RICA.”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, methodology): “The formulation of ICA in Equation 4 is called Reconstruction ICA [34].”
Learning hierarchical sparse features for RGB-(D) object recognition (2014, background): “Tiled convolutional neural networks (Le et al., 2011) are a type of sparse deep architecture with three important ingredi-ents: local receptive ﬁelds, pooling, and local contrast normalization that enjoy the beneﬁt of signiﬁcantly reducing the number of learnable parameters while giving the…”
Score Function Features for Discriminative Learning: Matrix and Tensor Framework (2014, background): “Instead, the recent trend has been to automatically learn good features through various frameworks such as deep learning (Bengio et al., 2013), sparse coding (Raina et al., 2007), independent component analysis (ICA) (Le et al., 2011), Fisher kernels (Jaakkola et al., 1999), and so on.”
Exemplar based Deep Discriminative and Shareable Feature Learning for scene image classification (2015, methodology): “Specifically, Random(3) feature, RICA [4] feature and our DDSFL features were extracted with step size 3 for the first layer, and step size 6 for the second and the third layer.”
Foundations and Advances in Deep Learning (2014, methodology): “This could be worked around by using regularization as in, for instance, sparse coding (Olshausen and Field, 1996) or independent component analysis (ICA) with reconstruction cost (Le et al., 2011b).”
Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition (2015, methodology): “[23] proposed amulti-modal unsupervised feature learning for RGB-D image classification based on [31].”
Real-time Neuroimaging and Cognitive Monitoring Using Wearable Dry EEG (2015, neutral): “This includes rejecting a subspace of ICA components precomputed using an (possibly overcomplete) decomposition [25] on calibration data or adaptively estimated using online recursive ICA [26], [27].”
Start from Scratch: Towards Automatically Identifying, Modeling, and Naming Visual Attributes (2014, methodology): “Different from traditional ICA [15] that optimizes a l1-norm penalty with “hard” orthonormality constraints, we use a “soft” version with the reconstruction cost [18], which meets our formulation in Eq.”
Direct Modeling of Complex Invariances for Visual Object Features (2013, background): “Could the established findings from invariance learning studies that exploit innate data characteristic, such as multi-way locality (Boureau et al., 2011) or its innate sparsity (Le et al., 2010; 2011; 2012), help in selecting the best complex invariant features?”
Analyzing noise in autoencoders and deep networks (2014, methodology): “A similar type of penalty is found when learning overcomplete representations with reconstruction ICA, where they use ‖WWT − I‖(2)F to encourage a diverse set of filters [14].”
Unsupervised Multilayer Feature Learning for Satellite Image Scene Classification (2016, background): “For potentially mining the complete characteristic of the original data, Zhang et al. [14] utilized a sparse autoencoder (S-AE) to implement the local convolution operation.”
Deep Learning of Invariant Features via Simulated Fixations in Video (2012, methodology): “The second term in Equation 1 is from the Reconstruction ICA a lgorithm [31].”