A Generalizable Approach to Learning Optimizers

By Diogo Almeida, Clemens Winter, Jie Tang, Wojciech Zaremba. 2021

35 citations recorded by the source corpus.

A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alexei A. Efros
2. Sergey Levine
3. Jitendra Malik
4. Trevor Darrell
5. Yoshua Bengio
6. Zhangyang Wang
7. Luc Van Gool
8. Quoc V. Le

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Jitendra Malik
4. Dacheng Tao
5. Trevor Darrell
6. Zhangyang Wang
7. Xiaogang Wang
8. Alexei A. Efros

In Optimization & Training

1. C. Freeman
2. Luke Metz
3. Niru Maheswaranathan
4. Wuyang Chen
5. Jascha Narain Sohl-Dickstein
6. Jitendra Malik
7. Jialin Liu
8. Alexei A. Efros

In Meta & Continual Learning

1. Jascha Narain Sohl-Dickstein
2. Niru Maheswaranathan
3. Luke Metz
4. C. Freeman
5. Alexei A. Efros
6. James Harrison
7. Tianlong Chen
8. Zhangyang Wang

Citations in the corpus

• Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017; 2019; Almeida et al., 2021).”
• Learning to Optimize: A Primer and A Benchmark (2021, background): “[11] learned to update optimizer hyperparameters instead of model parameters, directly using novel features, actions, and a reward function to feed RL.”
• Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “…including hand-designed optimizers (Bottou, 2010; Tieleman and Hinton, 2012; Kingma and Ba, 2014), learned hyper-parameter controllers (Daniel et al., 2016; Hansen, 2016; Xu et al., 2017; 2019; Almeida et al., 2021), and neural network based learned optimizers (Andrychowicz et al., 2016; Wichrowska”