Self-taught learning: transfer learning from unlabeled data

By Rajat Raina, Alexis Battle, Honglak Lee, Ben Packer, Andrew Y. Ng. 2007

1,809 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (2013, background): “Transfer learning across tasks using deep representations has been extensively studied, especially in an unsupervised setting (Raina et al., 2007; Mesnil et al., 2012).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Pretraining deep models has been studied extensively in the unsupervised setting (Hinton et al., 2006; Bengio et al., 2006; Vincent et al., 2008; Raina et al., 2007; Mesnil et al., 2012), in the supervised setting was perhaps popularized by ImageNet (Deng et al.”
Building high-level features using large scale unsupervised learning (2011, background): “Using unlabeled data in the wild to learn features is the key idea behind the self-taught learning framework (Raina et al., 2007).”
Multimodal Deep Learning (2011, background): “The multimodal learning settings we consider can be viewed as a special case of self-taught learning [4].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “fer methods pre-train effective feature representations to pre-encode knowledge across domains and tasks (Johnson and Zhang, 2005; Evgeniou and Pontil, 2007; Dai et al., 2007; Raina et al., 2007).”
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, neutral): “We remark that our setup on unlabeled data is similar to self-taught learning [31] rather than semi-supervised learning, because we do not assume any correlation between unlabeled data and the labeled training dataset.”
Class-incremental Learning via Deep Model Consolidation (2019, methodology): “Our work shares a similar spirit with self-taught learning [41], where we use unlabeled auxiliary data but do not require the auxiliary data to have the same class labels or generative distribution as the target data.”
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis (2019, methodology): “While we initialized SLDA using the standard base initialization procedure used by iCaRL and others, the covariance matrix could instead be initialized using large amounts of unlabeled imagery (i.e., self-taught learning [42]).”
Unsupervised Representation Learning by Sorting Sequences (2017, background): “A new unsupervised learning paradigm has recently emerged as self-supervised learning [32, 42].”
Lifelong Learning for Sentiment Classification (2015, neutral): “…learning (Jiang, 2008, Pan and Yang, 2010), multi-task learning (Caruana, 1997), never-ending learning (Carlson et al., 2010), self-taught learning (Raina et al., 2007), and online learning (Bottou, 1998), there is still no uniﬁed deﬁnition for LL. Based on the prior work and our research, to…”
Deep Learning of Representations: Looking Forward (2013, background): “Sparse coding is not actually very good as a generative model, but has been very successful for unsupervised feature learning (Raina et al., 2007; Coates and Ng, 2011; Yu et al., 2011; Grosse et al., 2007; Jenatton et al., 2009; Bach et al., 2011).”
Causal Transfer in Machine Learning (2015, methodology): “An alternative family of methods is based on learning a set of common features for all tasks [Argyriou et al., 2007a, Romera-Paredes et al., 2012, Argyriou et al., 2007b, Raina et al., 2007].”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, background): “Different from most transfer learning methods (Raina et al. 2007; Long et al. 2015; 2016), our algorithm leverages the dual structure of machine translation and achieves knowledge transfer through data sampling.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “DA is rather different also from self-taught learning [172], which exploits a limited labeled target data for a classiﬁcation task together with a large amount of unlabeled source data mildly related to the task.”
Sparse coding for multitask and transfer learning (2012, background): “e of the ideas presented here in the case of inﬁnite sample size. The second paper provides a learning to learn analysis of the multitask feature learning method in [3]. There are other works such as [19, 27] which have explored the application of sparse coding for supervised learning. The main idea”
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding (2012, methodology): “Examples of shallow models applied to feature discovery include sparse coding (Raina et al., 2007), restricted Boltzmann machines (RBMs) (Hinton et al., 2006; Courville et al., 2011b), various autoencoderbased models (Bengio et al., 2007), and hybrids of autoencoders and sparse coding (Kavukcuoglu…”
Selfie: Self-supervised Pretraining for Image Embedding (2019, methodology): “Although self-supervised/unsupervised representation learning (Hinton et al., 2006; Bengio et al., 2007; Raina et al., 2007; Vincent et al., 2010) was attempted to address this weakness, most practical neural network systems today are trained with supervised learning (Hannun et al., 2014; He et…”