Exploring loss function topology with cyclical learning rates
By Leslie N. Smith, Nicholay Topin. 2017
27 citations recorded by the source corpus.
We present observations and discussion of previously unreported phenomena discovered while training residual networks. The goal of this work is to better understand the nature of neural networks through the examination of these new empirical results. These behaviors were identified through the application of Cyclical Learning Rates (CLR) and linear network interpolation. Among these behaviors are counterintuitive increases and decreases in training loss and instances of rapid training. For example, we demonstrate how CLR can produce greater testing accuracy than traditional training despite using large learning rates. Files to replicate these results are available at this https URL
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Andrew Gordon Wilson
- Pavel Izmailov
- Dmitry P. Vetrov
- Timur Garipov
- Zheng Xu
- Tom Goldstein
- Dmitrii Podoprikhin
- Gavin Taylor
Citations in the corpus
- Visualizing the Loss Landscape of Neural Nets (2017, background): “Smith and Topin [34] use the same technique to show different minima and the “peaks” between them, while Im et al.”
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018, methodology): “Cyclical learning rates have also recently been considered in Smith and Topin [20] and Huang et al.”
- Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train (2017, unknown): “Based on our own empirical results and inspired by research such as the Cyclic learning rate [8] and SGDR [24], we believe that as far as SGD based methods are concerned, there is a correlation between a measure of regularity of a dataset, learning rate policy, and maximum global batch size.”