Workshop on Unsupervised and Transfer Learning Deep Learning of Representations for Unsupervised and Transfer Learning

By Yoshua Bengio. 2011

1,385 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

How transferable are features in deep neural networks? (2014, background): “We are interested in the answers to these questions because, to the extent that features within a network are general, we will be able to use them for transfer learning (Caruana, 1995; Bengio et al., 2011; Bengio, 2011).”
Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, background): “Multiple application domains have typically been studied in the context of transfer learning [14].”
Representation Learning: A Review and New Perspectives (2012, background): “An interesting hypothesis emerges to answer that question, out of recent theoretical results (Vincent, 2011; Alain and Bengio, 2012): the training criterion of regularized auto-encoders, instead of being a form of maximum likelihood, corresponds to a different inductive principle, such as score…”
Progressive Neural Networks (2016, methodology): “We report results by averaging the top 3 out of 25 jobs, each having different seeds and random hyper-parameter sampling.”
Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm (2017, methodology): “Instead we use transfer learning (Bengio et al., 2012) as described in § 3.3, which does not require access to the original dataset, but only the pretrained classiﬁer.”
Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, background): ““RBMs are tricky because although there are good estimators of the log-likelihood gradient, there are no known cheap ways of estimating the log-likelihood itself" [68].”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, background): “ poisoning attacks [20], and trojaned models [25], [30]. Training a large neural network from scratch requires a huge amount of data and computation, and developers tend to reuse pre-trained networks [3]. Reusing a pre-trained neural network exposes a DL system to several attacks. In particular, we ”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “2) Transfer Learning With Deep Models: Transfer learning, which falls in the area of domain adaptation and multi-task learning, involves the adaptation and improvement of learning in a new domain by transferring the knowledge representation that has been learned from data of a related domain [62].”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, neutral): “Then, the model can be fine-tuned [8, 6, 22].”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Knowledge transfer techniques like distillation (Bengio, 2012; Rusu et al., 2016; Parisotto et al., 2016; Teh et al., 2017) have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task…”
Deep Learning for Financial Applications : A Survey (2020, background): ““RBMs are tricky because although there are good estimators of the log-likelihood gradient, there are no known cheap ways of estimating the log-likelihood itself" [30].”
Progress & Compress: A scalable framework for continual learning (2018, background): “Early successful applications include unsupervised to supervised transfer learning (Bengio, 2012) and various results in the vision domain.”
GAN Memory with No Forgetting (2020, methodology): “Distinct from existing methods, our GAN memory leverages transfer learning [6, 16, 46, 92, 58] and (image) style transfer [18, 30, 41].”
Distral: Robust multitask reinforcement learning (2017, background): “Indeed this intuition has long been pursued in the multitask and transfer-learning literature [3, 27, 30, 6].”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “The optimal learning rate is usually close to the largest learning rate that does not cause divergence of the training criterion, an observation that can guide heuristics for setting the learning rate (Bengio, 2011), e.”
Meta-Learning: A Survey (2018, neutral): “…suitable for transfer learning because both the structure and the model parameters of the source models can be used as a good initialization for the target model, yielding a pre-trained model which can then be further ﬁne-tuned using the available training data on t new [168, 11, 13, 24].”
FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation (2018, background): “The early works (Caruana, 1995; Bengio, 2012; Donahue et al., 2014) apply transfer learning methods to ﬁnetune pre-trained models from the common classes containing adequate instances to the uncommon classes with only few instances.”