Exploring Strategies for Training Deep Neural Networks

By Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, Pascal Lamblin. 2009

1,259 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “The implementation is also reusable: it may be used for model fine-tuning and unsupervised pre-training [43, 45], which are forms of transfer learning, in which the parameters of a model trained on one task (e.”
Why Should We Add Early Exits to Neural Networks? (2020, background): “Note that supervised and unsupervised LT methods were explored repeatedly in the initial attempts at building deeper architectures [10, 35].”
Representation Learning: A Review and New Perspectives (2012, background): “…was empirically observed that layerwise stacking of feature extraction often yielded better representations, e.g., in terms of classification error (Larochelle et al., 2009; Erhan et al., 2010b), quality of the samples generated by a probabilistic model (Salakhutdinov and Hinton, 2009) or in terms…”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, background): “These gradients may push the generative model towards a poor local minimum— previous work has argued that deep neural networks (which form the conditional probability distributions p θ ( x | z )) are often sensitive to initialization (Glorot & Bengio, 2010; Larochelle et al. , 2009).”
The Limitations of Deep Learning in Adversarial Settings (2015, background): “However such solutions are hard to implement for deep neural networks because of non-convexity and non-linearity [25].”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “This kind of AE can be trained either layer by layer, as several shallow stacked AEs, or as a deep ANN [52].”
Opening the Black Box of Deep Neural Networks via Information (2017, neutral): “This looks very much like overﬁtting the small sample noise, which can be avoided with early stopping methods [Larochelle et al. (2009)].”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “For instance, genetic algorithms (GA) have been applied to evolve increasingly complex neural network topologies and the weights simultaneously, in the so-called NeuroEvolution of Augmenting Topologies (NEAT) method [25, 35].”
Deep learning for financial time series forecasting in A-Trader system (2017, neutral): “[30,31].”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “for early stopping as well), as in (Larochelle et al., 2009).”
Manitest: Are classifiers really invariant? (2015, background): “Another approach for measuring invariance consists in generating datasets with transformed images, and measuring the accuracy of classiﬁers on these datasets [19, 21, 31].”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “…3: Directly training all the layers of a deep network together not only makes it diﬃcult to exploit all the extra modeling power of a deeper architecture but in many cases it actually yields worse results as the number of required layers is increased (Larochelle et al., 2009; Erhan et al., 2010).”
Learning with Hierarchical-Deep Models (2013, background): “Index Terms—Deep networks, deep Boltzmann machines, hierarchical Bayesian models, one-shot learning Ç”
Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records (2016, methodology): “We used the same parameters in all the autoencoders of the deep architecture (regardless the layer) since this configuration usually leads to similar performances as having different parameters for each layer and is easier to evaluate 18,31 .”
Improved Contact Predictions Using the Recognition of Protein Like Contact Patterns (2014, background): “Deep learning architectures based on neural networks have some drawbacks: for example they are computationally demanding and require careful tuning [46].”
Maximum Entropy Generators for Energy-Based Models (2019, background): “The early work on deep learning relied on unsupervised learning (Hinton et al., 2006; Bengio et al., 2007; Larochelle et al., 2009) to train energy-based models (LeCun et al., 2006), in particular Restricted Boltzmann Machines, or 1 Montr´eal Institute for Learning Algorithms (Mila) 2 Canadian…”