Unsupervised and Transfer Learning Challenge: a Deep Learning Approach

By Grégoire Mesnil, Yann Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, Ian J. Goodfellow, Erick Lavoie, Xavier Muller, Guillaume Desjardins, David Warde-Farley and 3 more. 2011

241 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (2013, background): “Transfer learning across tasks using deep representations has been extensively studied, especially in an unsupervised setting (Raina et al., 2007; Mesnil et al., 2012).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Pretraining deep models has been studied extensively in the unsupervised setting (Hinton et al., 2006; Bengio et al., 2006; Vincent et al., 2008; Raina et al., 2007; Mesnil et al., 2012), in the supervised setting was perhaps popularized by ImageNet (Deng et al.”
Representation Learning: A Review and New Perspectives (2012, methodology): “The DAE and CAE have been successfully used to win the final phase of the Unsupervised and Transfer Learning Challenge (Mesnil et al., 2011).”
Progressive Neural Networks (2016, methodology): “We report results by averaging the top 3 out of 25 jobs, each having different seeds and random hyper-parameter sampling.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “The idea, described by Mesnil et al. (2011); Bengio (2011), is to perform greedy choices for the hyper-parameters associated with lower layers (near the input) before training the higher layers.”
From generic to specific deep representations for visual recognition (2014, background): “The ability of deep network’s representation has been also studied from various perspectives [4] including transfer learning [18] and domain adaptation [28].”
Deeper Insights into Weight Sharing in Neural Architecture Search (2020, methodology): “Inspired by the great success of transfer learning (Caruana, 1995; Mesnil et al., 2011; Kornblith et al., 2019), we try to do a similar thing by sharing one copy of “backbone” network while keeping a separated copy of “head” for each child model.”
Deep learning for visual understanding (2017, unknown): “Both DAE and CAE have been successfully used in the Unsupervised and Transfer Learning Challenge [119].”
Adversarial Reprogramming of Neural Networks (2018, background): “Transfer learning is a well studied topic in machine learning [24, 25].”
Towards Principled Unsupervised Learning (2015, methodology): “These ideas have also been used for semi-supervised learning (Kingma et al., 2014; Rasmus et al., 2015) and for transfer learning (Mesnil et al., 2012).”
Learning Categories From Few Examples With Multi Model Knowledge Transfer (2014, unknown): “Moreover, the feature transfer approach has proven to be extremely useful in the deep learning framework for unsupervised classiﬁcation tasks [23].”
General Stochastic Networks for Classification (2014, background): “This might also open a new prospect in the field of transfer learning [19] and multi-task learning [20] beyond classification.”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “Another widely-used technique is ﬁnetuning: a model is pretrained on a source domain and the output layers are ﬁnetuned via backpropagation in the target domain (Hinton and Salakhutdinov, 2006; Mesnil et al., 2012).”