Using Deep Q-Learning to Control Optimization Hyperparameters

By Samantha Hansen. 2016

45 citations recorded by the source corpus.

We present a novel definition of the reinforcement learning state, actions and reward function that allows a deep Q-network (DQN) to learn to control an optimization hyperparameter. Using Q-learning with experience replay, we train two DQNs to accept a state representation of an objective function as input and output the expected discounted return of rewards, or q-values, connected to the actions of either adjusting the learning rate or leaving it unchanged. The two DQNs learn a policy similar to a line search, but differ in the number of allowed actions. The trained DQNs in combination with a gradient-based update routine form the basis of the Q-gradient descent algorithms. To demonstrate the viability of this framework, we show that the DQN's q-values associated with optimal action converge and that the Q-gradient descent algorithms outperform gradient descent with an Armijo or nonmonotone line search. Unlike traditional optimization methods, Q-gradient descent can incorporate any objective statistic and by varying the actions we gain insight into the type of learning rate adjustment strategies that are successful for neural network optimization.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Learning to Optimize Neural Nets (2017, background): “Some rules for adjusting hyperparameters are manually designed (Bray et al., 2004), while others learn policies for adjusting the step size (Hansen, 2016; Daniel et al., 2016; Fu et al., 2016) or the damping factor in the Levenberg-Marquardt algorithm (Ruvolo et al., 2009).”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “…including hand-designed optimizers (Bottou, 2010; Tieleman and Hinton, 2012; Kingma and Ba, 2014), learned hyper-parameter controllers (Daniel et al., 2016; Hansen, 2016; Xu et al., 2017; 2019; Almeida et al., 2021), and neural network based learned optimizers (Andrychowicz et al., 2016; Wichrowska”
Deep Q-Networks for Accelerating the Training of Deep Neural Networks (2016, methodology): “For tuning learning rates, the most similar work is [3], where a DQN has been proposed to control them.”
Learning Gradient Descent: Better Generalization and Longer Horizons (2017, background): “Two recent papers (Daniel et al., 2016; Hansen, 2016) trained adaptive controllers to adjust the hyperparameters (learning rate) of traditional optimization algorithms from this perspective.”
Probabilistic Line Searches for Stochastic Optimization (2015, methodology): “m one gradient descent step to another. Several very recent works have proposed the use of reinforcement learning and ‘learning-to-learn’ approaches for parameter adaption (Andrychowicz et al., 2016; Hansen, 2016; Li and Malik, 2016). Mostly these methods are designed to work well on a specied subse”