Learning to Optimize Neural Nets

By Ke Li, Jitendra Malik. 2017

136 citations recorded by the source corpus.

Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “The concept of having one neural network interact with a second neural network to aid the learning process, or learning to learn or meta-learning [23, 49] has attracted much attention in recent years [1, 62, 14, 19, 35, 45, 15].”
Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning (2020, neutral): “To make the model sensitive to the new task, one popular direction of meta-learning is to train a meta-learner to learn how to update the parameters of the underlying model [Li and Malik, 2017; Ha et al. , 2017], which has been investigated in MAML [Finn et al. , 2017].”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “This goal is shared with recently-proposed methods by Andrychowicz et al. (2016); Ravi & Larochelle (2017); Wichrowska et al. (2017); Li & Malik (2016; 2017), which learn to generate numerical updates for training models.”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “Meta-learning has also recently been implemented with learned update rules ( e.g., Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Li & Malik, 2017a; Ha et al., 2017)) and memory-augmentation ( e.g., Santoro et al., 2016)).”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, methodology): “1 Several versions of this approach, i.e. Ravi & Larochelle (2017); Li & Malik (2017b), have the recurrent meta-learner operate on order-invariant features such as the gradient and objective value averaged over the datapoints in the dataset, rather than operating on the individual datapoints…”
Meta-Learning: A Survey (2018, neutral): “Follow-up work (Li and Malik, 2017) shows how to leverage this approach to learn optimization algorithms for (shallow) neural networks.”
Natural Language to Structured Query Generation via Meta-Learning (2018, methodology): “This direction has been applied to learning to optimize deep neural networks (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li and Malik, 2017; Ha et al., 2017).”
Truncated Back-propagation for Bilevel Optimization (2018, background): “By contrast, treating A λ as a general function approximator relies on the availability of a large amount of meta training data [21, 22].”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, methodology): “…Mishra et al., 2017), methods that learn an optimizer and/or initialization (Andrychowicz et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017a; Li & Malik, 2017), and methods that compare new datapoints in a learned metric space (Koch, 2015; Vinyals et al., 2016; Shyam et al., 2017; Snell et…”
Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon (2018, background): “…is worth mentioning that learning recurrent algorithmic decisions is also used in the deep learning community, for instance in the ﬁeld of meta-learning to decide how to apply gradient updates in stochastic gradient descent (Andrychowicz et al., 2016; Li and Malik, 2017; Wichrowska et al., 2017).”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, methodology): “Prior work has proposed memory-based methods [5, 30, 22, 26] and methods that learn an optimizer and/or a parameter initialization [3, 20, 9].”
Understanding and correcting pathologies in the training of learned optimizers (2018, methodology): “On one hand are black-box meth-ods such as evolutionary algorithms (Goldberg & Holland, 1988; Bengio et al., 1992), random search (Bergstra & Ben-gio, 2012), reinforcement learning (Bello et al., 2017; Li & Malik, 2017a;b), or Bayesian optimization (Snoek et al., 2012).”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “…techniques have included gradient descent (Maclaurin et al., 2015; Andrychowicz et al., 2016; Wichrowska et al., 2017), reinforcement learning (Li and Malik, 2017a;b), and more advanced training procedures (Lv et al., 2017; Maheswaranathan et al., 2019; Metz et al., 2019a; Chen et al., 2020;…”