Greedy Layer-Wise Training of Deep Networks

By Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle. 2006

5,483 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “reeze additional layers for further ﬁne-tuning. The simplest approach to treating different layers appropriately is to ﬁne-tune the model one layer at a time, analogous to greedy layer-wise training (Bengio et al., 2007) and ‘chain-thaw’ (Felbo et al., 2017). However, this introduces a sequential re”
Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model (2015, neutral): “The last two years, huge improvements have been observed on this task thanks to the recent advances of deep learning community [18, 1, 14].”
FitNets: Hints for Thin Deep Nets (2014, background): “On the one hand, pre-training strategies, whether unsupervised (Hinton et al., 2006; Bengio et al., 2007) or supervised (Bengio et al., 2007) train the network parameters in a greedy lay-erwise fashion in order to initialize the network parameters in a potentially good basin of attraction.”
Deeply-Supervised Nets (2014, methodology): “To investigate the difference between DSN and layer-wise pre-training (shown as CNN with pre-training in Table 1) , we train a CNN model using greedy layer-wise pre-training method as in [1].”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, methodology): “Next, we apply these solutions to investigate the commonly used greedy layer-wise pretraining strategy for training deep networks [17, 18], and recover conditions under which such pretraining speeds learning.”
Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “…for reviews) grew from breakthroughs in unsupervised learning of representations, based mostly on the Restricted Boltzmann Machine (RBM) (Hinton et al., 2006), auto-encoder variants (Bengio et al., 2007; Vin-cent et al., 2008), and sparse coding variants (Lee et al., 2007; Ranzato et al., 2007).”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “n with the beneﬁts of the additional language modeling objectives and residual connections. 3 RELATED WORK Pretraining was once an essential technique for training deep networks (Hinton et al., 2006; Bengio et al., 2007; Dahl et al., 2010; Erhan et al., 2010). However, better optimization techniques”
Fine-tuned Language Models for Text Classification (2018, background): “The simplest approach to treating different layers appropriately is to ﬁne-tune the model one layer at a time, analogous to greedy layer-wise training (Bengio et al., 2007) and ‘ chain-thaw ’ (Felbo et al., 2017).”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, neutral): “One approach to semi-supervised learning involves unsupervised or self-supervised pretraining, followed by supervised fine-tuning [3, 4].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, methodology): “In that sense our approach resembles layer-wise training of autoencoders (Bengio et al., 2007).”
GSNs : Generative Stochastic Networks (2015, background): “Information and Inference: A Journal of the IMA (2016) Page 1 of 40 doi:10.1093/imaiai/iaw003 GSNs: generative stochastic networks”
Pedestrian Detection with Unsupervised Multi-stage Feature Learning (2012, background): “In [16], it was shown that unsupervised learning can be used to train deep hierarchical models and the final representation achieved is actually useful for a variety of different tasks [32, 24, 4].”
From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, background): “Deep learning architectures, such as deep neural networks (Hinton & Salakhutdinov, 2006), stacked auto-encoders (Bengio et al., 2007; Vincent et al., 2008), or convolutional neural networks (LeCun et al., 1998), are the current state of the art in learning parsimonious representations of…”
Generative Moment Matching Networks (2015, neutral): “Greedy layer-wise pretraining of the auto-encoder (Bengio et al., 2007).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Pretraining deep models has been studied extensively in the unsupervised setting (Hinton et al., 2006; Bengio et al., 2006; Vincent et al., 2008; Raina et al., 2007; Mesnil et al., 2012), in the supervised setting was perhaps popularized by ImageNet (Deng et al.”