Learned Optimizers that Scale and Generalize

By Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, Jascha Narain Sohl-Dickstein. 2017

308 citations recorded by the source corpus.

Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks, and which has significantly reduced memory and computation overhead. We achieve this by introducing a novel hierarchical RNN architecture, with minimal per-parameter overhead, augmented with additional architectural features that mirror the known structure of optimization tasks. We also develop a meta-training ensemble of small, diverse, optimization tasks capturing common properties of loss landscapes. The optimizer learns to outperform RMSProp/ADAM on problems in this corpus. More importantly, it performs comparably or better when applied to small convolutional neural networks, despite seeing no neural networks in its meta-training set. Finally, it generalizes to train Inception V3 and ResNet V2 architectures on the ImageNet dataset for thousands of steps, optimization problems that are of a vastly different scale than those it was trained on.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “An exception is the recent work focused on learning an optimizer for ImageNet classification that achieved notable improvements [64].”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “This goal is shared with recently-proposed methods by Andrychowicz et al. (2016); Ravi & Larochelle (2017); Wichrowska et al. (2017); Li & Malik (2016; 2017), which learn to generate numerical updates for training models.”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “tures, such as a second neural network [73, 60], an LSTM [67], a hierarchical RNN [88], or even a different LSTM for each weight [53].”
Scalable Neural Architecture Search for 3D Medical Image Segmentation (2019, neutral): “t of automated machine learning (AutoML). Yet, in contrast to the recent studies on the use of advanced AutoML algorithms such as neural architecture search (NAS) [5,9,12] and neural optimizer search [10] for general computer vision tasks, only few approaches using simple hyperparameter optimization”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, methodology): “RL-based neural architecture search yielded success in automatic design of state-of-the-art RNN cells [50], convolutional blocks [51], activation functions [32], optimizers [3, 42] and data augmentation strategies [10].”
Transfer Learning with Neural AutoML (2018, methodology): “These include random search [1], parameter modeling [3], metalearned hyperparameter initialization [16], deep-learning based tree searches over a predefined model-specification language [17], and learning of gradient descent optimizers [18, 19].”
Meta-Gradient Reinforcement Learning (2018, methodology): “A recent direction of research has been to meta-learn an optimiser using a recurrent parameterisation [Andrychowicz et al., 2016, Wichrowska et al., 2017].”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, neutral): “A variety of meta-learners that employ gradient methods for task-speciﬁc fast adaptation have been proposed ( e.g., Andrychowicz et al., 2016; Li & Malik, 2017a;b; Wichrowska et al., 2017).”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, background): “While these models are very powerful, they have trouble scaling to deep models with a large number of parameters (Wichrowska et al., 2017) without tricks to simplify the formulation.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “When it is computed as the sum of the validation loss after each inner optimization step, then meta-training also encourages faster learning in the base task [78], [91], [92].”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, methodology): “(6) is gradient descent with momentum; in (Andrychowicz et al., 2016; Wichrowska et al., 2017) the mapping Φ t is implemented as a recurrent neural network, while (Finn et al., 2017) focus on the initialization mapping by letting Φ 0 ( λ ) = λ .”
Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon (2018, background): “…is worth mentioning that learning recurrent algorithmic decisions is also used in the deep learning community, for instance in the ﬁeld of meta-learning to decide how to apply gradient updates in stochastic gradient descent (Andrychowicz et al., 2016; Li and Malik, 2017; Wichrowska et al., 2017).”
How to Train Your MAML to Excel in Few-Shot Classification (2021, background): “Recent years have witnessed an abundance of new approaches on meta-learning [29, 30, 48, 61, 63, 64, 68, 77, 79, 80, 94, 98], developed in various areas including few-shot learning [70, 85, 89, 105, 107, 110, 111, 115], optimization [3, 9, 53, 108], reinforcement and imitation learning [23, 24, 31, ”
Revisiting Meta-Learning as Supervised Learning (2020, background): “, 2017; Wang and Hebert, 2016), optimization (Li and Malik, 2017; Wichrowska et al., 2017), active learning (Bachman et al.”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “…et al., 2018), unsupervised learning (Gupta et al., 2018; Hsu et al., 2018), few-shot model adaptation (Nagabandi et al., 2018), and improving the optimizers (Andrychowicz et al., 2016; Li & Malik, 2017; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Chen et al., 2016; Gupta et al., 2018).”
Learning to learn by gradient descent by gradient descent (2019, methodology): “In this paper [3], they introduced a hierarchical RNN to add structural dependencies between parameters.”