Learning Gradient Descent: Better Generalization and Longer Horizons

By Kaifeng Lyu, Shunhua Jiang, Jian Li. 2017

120 citations recorded by the source corpus.

Training deep neural networks is a highly nontrivial task, involving carefully selecting appropriate training algorithms, scheduling step sizes and tuning other hyperparameters. Trying different combinations can be quite labor-intensive and time consuming. Recently, researchers have tried to use deep learning algorithms to exploit the landscape of the loss function of the training problem of interest, and learn how to optimize over it in an automatic way. In this paper, we propose a new learning-to-learn model and some useful and practical tricks. Our optimizer outperforms generic, hand-crafted optimization algorithms and state-of-the-art learning-to-learn optimizers by DeepMind in many tasks. We demonstrate the effectiveness of our algorithms on a number of tasks, including deep MLPs, CNNs, and simple LSTMs.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Understanding and correcting pathologies in the training of learned optimizers (2018, background): “On the other hand, a complementary approach is to learn the optimization algorithm (Bengio et al., 1990; Schmidhuber, 1995; Hochreiter et al., 2001; Andrychowicz et al., 2016; Wichrowska et al., 2017; Li & Malik, 2017b; Lv et al., 2017; Bello et al., 2017).”
Understanding Short-Horizon Bias in Stochastic Meta-Optimization (2018, background): “More ambitiously, others have attempted to learn an optimizer (Andrychowicz et al., 2016; Li & Malik, 2017; Finn et al., 2017; Lv et al., 2017; Wichrowska et al., 2017; Metz et al., 2017).”
Gradients are Not All You Need (2021, methodology): “This has been used to great effect in training of language models [Sutskever, 2013], learned optimizers [Andrychowicz et al., 2016, Wichrowska et al., 2017, Lv et al., 2017, Metz et al., 2019], and ﬂuid simulation [Kochkov et al., 2021].”
Learning to Optimize: A Primer and A Benchmark (2021, methodology): “L2O-RNNprop has slower convergence than ISTA/FISTA and only produces reasonably good solutions after 1,000 iterations – though still much better than analytic optimizers Adam and RMSProp.”
Bootstrapped Meta-Learning (2021, background): “IMPALA 192%, [8] Metagradient 287%, [44] STACX 364%, [45] LASER 431%, [32] BMG 611% (ours) BMG w.”
Meta Continual Learning (2018, methodology): “Whereas this variant of the backpropagation was formulated by human experts, some learning-to-learn or meta-learning approaches successfully designed optimizers for update rules in an automatic way [4, 13, 15].”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “, 2017), reinforcement learning (Li and Malik, 2017a;b), and more advanced training procedures (Lv et al., 2017; Maheswaranathan et al., 2019; Metz et al., 2019a; Chen et al., 2020; Vicol et al., 2021; Metz et al., 2021) leveraging both Evolution Strategies (ES), and gradients.”