Learning Step Size Controllers for Robust Neural Network Training

By Christian Daniel, Jonathan Taylor, Sebastian Nowozin. 2016

96 citations recorded by the source corpus.

This paper investigates algorithms to automatically adapt the learning rate of neural networks (NNs). Starting with stochastic gradient descent, a large variety of learning methods has been proposed for the NN setting. However, these methods are usually sensitive to the initial learning rate which has to be chosen by the experimenter. We investigate several features and show how an adaptive controller can adjust the learning rate without prior knowledge of the learning problem at hand.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, background): “Some rules for adjusting hyperparameters are manually designed (Bray et al., 2004), while others learn policies for adjusting the step size (Hansen, 2016; Daniel et al., 2016; Fu et al., 2016) or the damping factor in the Levenberg-Marquardt algorithm (Ruvolo et al., 2009).”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017; 2019; Almeida et al., 2021).”
Meta-Learning: A Survey (2018, neutral): “, 1997) or the learning rate for gradient descent (Daniel et al., 2016) to the task at hand.”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, background): “…one particle is used in SVGD. Generative adversarial network (GAN) and its variants have recently gained remarkable success on generating realistic-looking images (Goodfellow et al., 2014; Salimans et al., 2016; Radford et al., 2015; Li et al., 2015; Dziugaite et al., 2015; Nowozin et al., 2016).”
A Generalizable Approach to Learning Optimizers (2021, background): “Most similar to our work are those that use reinforcement learning to control learning rate [12, 50, 51].”
Step Size Matters in Deep Learning (2018, background): “The effect of the step size on training neural networks was empirically investigated in (Daniel et al., 2016).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “This allows a policy to be learned (e.g., based on reinforcement learning) for mapping a state of the learner to an HPC, for example, the learning rate (Daniel et al., 2016).”
Interactive Machine Learning via a GPU-accelerated Toolkit (2017, methodology): “Besides hand tuning, researchers have developed auto tuning algorithms including random search [3], gradient based methods [26, 2], reinforcement learning [10] or bayesian optimization [34].”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “…optimizers, including hand-designed optimizers (Bottou, 2010; Tieleman and Hinton, 2012; Kingma and Ba, 2014), learned hyper-parameter controllers (Daniel et al., 2016; Hansen, 2016; Xu et al., 2017; 2019; Almeida et al., 2021), and neural network based learned optimizers (Andrychowicz et al.,…”
Deep Q-Networks for Accelerating the Training of Deep Neural Networks (2016, methodology): “ 4 DQN FOR TUNING LEARNING RATES 4.1 WEIGHTS FEATURES AS STATES In this paper, the states fed into a DQN are extracted from the weights of a DNN during training. This is different from Hansen (2016); Daniel et al. (2016), where state features are manually crafted by the author. This not only avoids ”
Reinforcement Learning for Learning Rate Control (2017, methodology): “Another recent work [ Daniel et al. , 2016 ] investigates several hand-tuned features and uses the Relative Entropy Policy Search method as controller to select step size for SGD and RMSprop.”
Learning an Adaptive Learning Rate Schedule (2019, methodology): “2 State Observation, Reward, Action Observation In order to characterize the training dynamics, we design a set of state observation features s, which is an extension of the features used in [9].”
Variants of RMSProp and Adagrad with Logarithmic Regret Bounds (2017, methodology): “RMSProp is one of the most popular adaptive gradient algorithms used for the training of deep neural networks (Schaul et al., 2014; Dauphin et al., 2015; Daniel et al., 2016; Schmidhuber, 2015).”
Proactive Resource Management for LTE in Unlicensed Spectrum: A Deep Learning Perspective (2017, methodology): “nterval of good learning rates at which the performance does not vary much [39]. This in turn necessitates the need for experimental studies in order to search for good problem-speciﬁc learning rates [48]. A typical range of the learning rate for the dataset under study falls approximately between 0”
Learning Gradient Descent: Better Generalization and Longer Horizons (2017, background): “In the case when the optimizee is stochastic, that is, the value of f(θ) de- pends on the sample d selected from a dataset D, the goal of an optimizer is to minimize 1 |D| ∑ d∈D fd(θ) (1) over the variables θ.”