Why Does Unsupervised Pre-training Help Deep Learning?

By Dumitru Erhan, Aaron C. Courville, Yoshua Bengio, Pascal Vincent. 2010

2,508 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Impact of LM fine-tuning We compare no finetuning against fine-tuning the full model (Erhan et al., 2010) (‘Full’), the most commonly used fine-tuning method, with and without discriminative fine-tuning (‘Discr’) and slanted triangular learning rates (‘Stlr’) in Table 6.”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “Indeed, many explanations for the difficulty of deep learning have been advanced in the literature, including the presence of many local minima, low curvature regions due to saturating nonlinearities, and exponential growth or decay of back-propagated gradients [12, 13, 14, 15].”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “guage modeling objectives and residual connections. 3 RELATED WORK Pretraining was once an essential technique for training deep networks (Hinton et al., 2006; Bengio et al., 2007; Dahl et al., 2010; Erhan et al., 2010). However, better optimization techniques and big labeled datasets rendered pretr”
Fine-tuned Language Models for Text Classification (2018, methodology): “Furthermore, when pretrained weights capturing certain properties of natural language are available, using random weights as initialization appears to be a suboptimal choice (Erhan et al., 2010).”
High-dimensional dynamics of generalization error in neural networks (2017, background): “11 of Erhan et al., 2010 for an empirical demonstration in a deep neural network).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “…corrupted versions, and is widely used as an effective self-supervised representation learning and pretraining strategy, e.g., in computer vision (Erhan et al., 2010) and natural language processing (NLP) (Lewis et al. (2020), which randomly drops / replaces tokens in a sentence and learns to…”
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, methodology): “The idea of using unlabeled data to pre-train neural networks has been shown effective across many fields, including computer vision (Vincent et al., 2008; Erhan et al., 2010), NLP (Peters et al.”
Deep Networks with Stochastic Depth (2016, methodology): “Earlier works adopted greedy layer-wise training or better initialization schemes to alleviate the vanishing gradients and diminishing feature reuse problems [12, 17, 18].”
Towards the Science of Security and Privacy in Machine Learning (2016, background): “Unsupervised learning considers problems such as clustering points according to a similarity metric [7], dimensionality reduction to project data in lower dimensional subspaces [8], and model pre-training [10].”
Representation Learning: A Review and New Perspectives (2012, methodology): “…that layerwise stacking of feature extraction often yielded better representations, e.g., in terms of classification error (Larochelle et al., 2009; Erhan et al., 2010b), quality of the samples generated by a probabilistic model (Salakhutdinov and Hinton, 2009) or in terms of the invariance…”
Stacked What-Where Auto-encoders (2015, background): “This has been recognized to generalize better since less freedom is given [3].”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “In several branches of machine learning, and in particular in deep learning, the use of related datasets in pre-training the model is an effective method of scaling up to complex environments (Erhan et al., 2010; Kumar et al., 2015).”
Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm (2017, methodology): “Alternatively, another common approach is to use the pretrained model as an initialization (Erhan et al., 2010), where the full model is unfrozen (hereafter referred to as ‘full’).”