Learning to Optimize

By Ke Li, Jitendra Malik. 2016

631 citations recorded by the source corpus.

Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm, which we believe to be the first method that can automatically discover a better algorithm. We approach this problem from a reinforcement learning perspective and represent any particular optimization algorithm as a policy. We learn an optimization algorithm using guided policy search and demonstrate that the resulting algorithm outperforms existing hand-engineered algorithms in terms of convergence speed and/or the final objective value.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “This approach has been applied to learning to optimize deep networks (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2017), as well as for learning dynamically changing recurrent networks (Ha et al., 2017).”
Learning to Optimize Neural Nets (2017, methodology): “In this paper, we build on the method proposed in (Li & Malik, 2016) and develop an extension that is suited to learning optimization algorithms for high-dimensional stochastic problems.”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “Another line of work (Hochreiter et al., 2001; Younger et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2016) studies meta-learning over the optimization process.”
One-Shot Imitation Learning (2017, neutral): “Meta-learning has also been studied to discover neural network weight optimization algorithms [8, 9, 23, 50, 2, 31].”
True Few-Shot Learning with Language Models (2021, neutral): “The meta-learning setting assumes access to data from many other tasks in order to perform learning and model selection [20, 21, 64, 65].”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, neutral): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Neural Architecture Search with Reinforcement Learning (2016, background): “More closely related is the idea of using a neural network to learn the gradient descent updates for another network (Andrychowicz et al., 2016) and the idea of using reinforcement learning to ﬁnd update policies for another network (Li & Malik, 2016).”
How to train your MAML (2018, background): “Recent work in meta-learning has produced state of the art results in a variety of settings (Wang et al., 2016; Ba et al., 2016; Zoph & Le, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016; Li & Malik, 2016; Li et al., 2017; Antoniou et al 2017; Brock et al., 2017; Munkhdalai & Yu, 2017).”
Learning Combinatorial Optimization Algorithms over Graphs (2017, methodology): “In continuous optimization, methods have been proposed for learning an update rule for gradient descent [2, 27] and solving black-box optimization problems [8]; these are very interesting ideas that highlight the possibilities for better algorithm design through learning.”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “A key advantage of our method of discovering update equations compared to the previous approaches (Andrychowicz et al., 2016; Li & Malik, 2016) is that update equations found by our method can be easily transferred to new tasks.”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, neutral): “Meta-learning: Our study is also related to meta-learning [5, 1, 20, 34].”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, neutral): “Li [6], and Andrychowicz [7] studied a LSTM based meta-learner that takes the gradients from learner and performs an optimization step.”
Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, background): “Meta-learners developed so far include recurrent models [6, 10, 19, 26], metrics [12, 25], or optimizers [2, 7, 16, 18].”
Meta Networks (2017, background): “There is a line of work on building meta optimizers (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2017).”
Learning to reinforcement learn (2016, background): “More recently, however, there has been a surge of interest in using neural networks to learn optimization procedures, using a range of innovative meta-learning techniques (Andrychowicz et al., 2016; Chen et al., 2016; Li and Malik, 2016; Zoph and Le, 2016).”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “Meta-learning has also recently been implemented with learned update rules ( e.g., Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Li & Malik, 2017a; Ha et al., 2017)) and memory-augmentation ( e.g., Santoro et al., 2016)).”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, neutral): “1 Several versions of this approach, i.e. Ravi & Larochelle (2017); Li & Malik (2017b), have the recurrent meta-learner operate on order-invariant features such as the gradient and objective value averaged over the datapoints in the dataset, rather than operating on the individual datapoints…”
Meta-Learning with Temporal Convolutions (2017, background): “ring data points is built into them. A number of methods consider a meta-learner that makes updates to the parameters of a traditional learner [ 10 , 4]. Both Andrychowicz et al. [1] and Li and Malik [16] investigated the setting of learning to optimize, where the learner is an objective function to”
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, background): “…for neural models that were explored in the past [32–34], and more recent approaches that focused on learning optimizers for deep networks [13, 35–37], generating model parameters [38– 40], learning task embeddings [14, 41] including memory-based approaches [12], learning to learn implicitly via…”