The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training

By Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, Pascal Vincent. 2009

472 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “As pointed by Erhan et al. (2009), pre-training strategies can act as regularizers.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “Tracing the intellectual origins of this idea even back further, the original inspiration comes from the computer vision community, dating back at least a decade [Erhan et al., 2009].”
Convergent Learning: Do different neural networks learn the same representations? (2015, background): “…Neyshabur & Panigrahy, 2013; Montavon et al., 2011; Paul & Venkatasubramanian, 2014; Goodfellow et al., 2014) and from an empirical perspective (Erhan et al., 2009; Eigen et al., 2013; Szegedy et al., 2013; Simonyan et al., 2013; Zeiler & Fergus, 2014; Nguyen et al., 2014; Yosinski et al.,…”
Knowledge Transfer in Deep Block-Modular Neural Networks (2015, background): “These successes can mainly be attributed to the use of pre-training [7,21], sharing [16], and various forms of regularization [8,10], as well as to increases in computational capacity and data availability [22].”
Krylov Subspace Descent for Deep Learning (2011, neutral): “This is opposite to the conclusions reache d in [3] (their conclusion was that pretraining helps by finding a better “basin of attraction”), bu t that paper was not using these types of optimization methods.”
Discriminative Neural Sentence Modeling by Tree-Based Convolution (2015, background): “However, they may have difficulties in learning deep dependencies because of long propagation paths (Erhan et al., 2009).”
Distributed optimization of deeply nested systems (2012, background): “The hidden units may encode hierarchical, distributed features that are useful to deal with complex sensory data.”
On Graduated Optimization for Stochastic Non-Convex Problems (2015, background): “In Bengio (2009), it is suggested to consider some developments in deep belief architectures Hinton et al. (2006); Erhan et al. (2009) as a kind of continuation.”
Neural conditional random fields (2010, background): “Due to non-convexity, initialization is a crucial step for NN learning, especially in the case of deep architectures (see (Erhan et al., 2009) for an analysis).”
Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis (2016, background): “4http://www.yelp.com/dataset challenge 5http://jmcauley.ucsd.edu/data/amazon/links.html entropy error, which is a common strategy for deep learning (Erhan et al., 2009).”
Feature space maximum a posteriori linear regression for adaptation of deep neural networks (2014, background): “Training a neural network directly from a set of randomly initialized parameters usually results in a poor local optimum when performing error backpropagation, especially when the neural network is deep [35].”