Practical Recommendations for Gradient-Based Training of Deep Architectures

By Yoshua Bengio. 2012

2,396 citations recorded by the source corpus.

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyperparameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “A detailed guide for setting hyper-parameters can be found in (Bengio, 2012).”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These first-order methods have been particularly successful for training highly-expressive, over-parameterized models such as non-parametric regression [7, 45] and deep neural networks [9, 88].”
Representation Learning: A Review and New Perspectives (2012, background): “more modern compendium of good training practice, particularly adapted to training RBMs, is provided in Hinton (2010), while a similar guide oriented more towards deep neural networks can be found in Bengio (2013), both of which are part of a novel version of the above book. Recent work on automatin”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “6 The best hyperparameters of the models were chosen by early stopping with patience on the validation set perplexity (Bengio 2012).”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Bengio (2012) suggests that it is often the single most important hyper-parameter and that it always should be tuned.”
Recent advances in convolutional neural networks (2015, methodology): “Next, we discuss some typical applications of CNN including image classification, object detection, object tracking, pose estimation, text detection and recognition, visual saliency detection, action recognition and scene labeling in Section V. Finally, we conclude this paper in Section VI.”
A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, methodology): “In practice, one should consider whether the best observed value falls near the border of the range searched over; if so, it is probably worth exploring beyond that border, as suggested in (Bengio, 2012).”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “In most cases, the LR must be manually adjusted during model training, and this adjustment is often necessary for enhanced accuracy (Bengio, 2012).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Training on 800 examples/10 epochs led to the repeated emergence of learning-rate decay, a wellknown strategy for the timely training of an ML model [9].”
Recognition in Terra Incognita (2018, background): “In this situation, it is common practice to employ early stopping [55] as a means of preventing overfitting to the train distribution.”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, background): “Bergstra and Bengio (2012) show that for most datasets only a few of the hyperparameters really matter.”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “Experience replay also breaks temporal correlations of samples used for Q-value updates—crucial for reducing generalization error, as the stochastic optimization algorithms used for training DQNs typically assume i.i.d. data (Bengio, 2012).”
A survey on deep learning in medical image analysis (2017, background): “Most researchers fall back to an intuition-based random search (Bergstra and Bengio, 2012), which often seems to work well enough.”
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning (2014, background): “Bengio (2013); Bengio et al. (2013a) have proposed the notion of conditional computation for deep learning to answer positively to this question.”
Troubling Trends in Machine Learning Scholarship (2018, background): “roduction, a designated “Motivation” section clearly quarantines this discussion. This practice avoids confusing readers while allowing authors to express informal ideas. In another positive example, [3] presents practical guidelines for training neural networks. Here, the authors carefully convey u”
Cyclical Learning Rates for Training Neural Networks (2015, methodology): “…rule of thumb that the optimum learning rate is usually within a factor of two of the largest one that converges [2] and set base lr to 13 or 1 4 of max lr. Figure 3 shows an example of making this type of run with the CIFAR-10 dataset, using the architecture and hyper-parameters provided by Caffe.”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “During training, we compute gradients on an entire minibatch at a time, allowing the use of matrix-matrix multiplications instead of matrix-vector multiplications (Bengio, 2012).”
Deep Learning of Representations: Looking Forward (2013, background): “A more generic way to avoid gradient vanishing is to incorporate a training penalty that encourages the propagated gradient vectors to maintain their magnitude (Pascanu and Bengio, 2012).”
Knowledge Matters: Importance of Prior Information for Optimization (2013, methodology): “The experiments performed also provide evidence in favor of the “ Deeper Harder Hypothesis ” and associated “ Abstractions Harder Hypothesis ”.”