A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay

By Leslie N. Smith. 2018

1,200 citations recorded by the source corpus.

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Metaparameter tuning is a crucial step of the deep learning pipeline (Bergstra and Bengio, 2012; Snoek et al., 2012; Sutskever et al., 2013; Smith, 2018), so it is critical for papers studying optimizers to match as closely as possible the tuning protocols of an ideal practitioner.”
UDSMProt: universal deep sequence models for protein classification (2019, methodology): “We use a 1-cycle learning rate schedule [33] during training for 30 epochs during the final finetuning step.”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, background): “2018) as an alternative to learning rate cooldown (Smith, 2018).”
Understanding Batch Normalization (2018, background): “Another line of work of relevance is [48] and [47], where the relationship between various network parameters, accuracy and convergence speed is investigated, the former article argues for the importance of batch normalization to facilitate a phenomenon dubbed ’super convergence’.”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, methodology): “For this ablation study we train PointPillars with 6x fewer epochs and a one cycle optimizer schedule [71] to cut down the training time.”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, methodology): “A way to further decrease the number of parameter updates is to use the one cycle policy whilst using the increasing batch size scheme [36].”
Frustratingly Easy Natural Question Answering (2019, methodology): “These parameters were tuned for each experiment using the approach outlined in (Smith 2018) where we evaluate a number of batch sizes and learning rates on a randomly selected 20% subset of the NQ training and development data.”
AI Feynman: A physics-inspired method for symbolic regression (2019, methodology): “The learning rate and momentum schedules were implemented as described in [26, 27] using the fastai package [28].”
Automated pavement crack detection and segmentation based on two‐step convolutional neural network (2020, methodology): “A cyclical learning rate schedule was used instead of a constant learning rate, which determines the ability to find the global minimum of the proposed method (Smith, 2017, 2018).”
One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision (2021, methodology): “To set the learning rate, we use the one-cycle policy [43] using the implementation in [23].”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, methodology): “Specifically, We trained all models for 40 epochs using Adam optimize and 1-cycle learning rate policy [73].”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “001 and is modified using the one-cycle learning rate policy [54].”
ParticleNet: Jet Tagging via Particle Clouds (2019, methodology): “[71], and slightly tuned afterward with a few trial trainings.”