Learning an Adaptive Learning Rate Schedule

By Zhen Xu, Andrew M. Dai, Jonas Kemp, Luke Metz. 2019

79 citations recorded by the source corpus.

The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In this paper, we propose a reinforcement learning based framework that can automatically learn an adaptive learning rate schedule by leveraging the information from past training histories. The learning rate dynamically changes based on the current training dynamics. To validate this framework, we conduct experiments with different neural network architectures on the Fashion MINIST and CIFAR10 datasets. Experimental results show that the auto-learned learning rate controller can achieve better test results. In addition, the trained controller network is generalizable -- able to be trained on one data set and transferred to new problems.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017; 2019; Almeida et al., 2021).”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, methodology): “…including hand-designed optimizers (Bottou, 2010; Tieleman and Hinton, 2012; Kingma and Ba, 2014), learned hyper-parameter controllers (Daniel et al., 2016; Hansen, 2016; Xu et al., 2017; 2019; Almeida et al., 2021), and neural network based learned optimizers (Andrychowicz et al., 2016; Wichrowska”
A Generalizable Approach to Learning Optimizers (2021, background): “Most similar to our work are those that use reinforcement learning to control learning rate [12, 50, 51].”
Reinforced Attention for Few-Shot Learning and Beyond (2021, background): “The applications are also studied on more classic tasks, such as classification [61], object detection [2, 35], segmentation [1, 66] and pose estimation [24, 44].”