Lookahead Optimizer: k steps forward, 1 step back

By Michael Ruogu Zhang, James Lucas, Geoffrey E. Hinton, Jimmy Ba. 2019

876 citations recorded by the source corpus.

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “For panoptic segmentation, we strictly follow Panoptic-DeepLab [20], except using a linear warm up Radam [61] Lookahead [96] optimizer (with the same learning rate 0.”
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “, 2019) (iv) Look-ahead optimizer (Zhang et al., 2019).”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “Most applications study models along the same optimization trajectory, e.g. (Ruppert, 1988; Polyak, 1990; Szegedy et al., 2016; Izmailov et al., 2018; Zhang et al., 2019; Kaddour et al., 2022; Junczys-Dowmunt et al., 2016).”
The Efficiency Misnomer (2021, background): “However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi-step lookahead schemes (Zhang et al., 2019).”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Specifically, we train our models with 32 TPU cores for 100k (400k for main results) iterations (54 epochs), a batch size of 64, Radam [62] Lookahead [104], a ‘poly’ schedule learning rate of 10 (3 ⇥ 10 for MaX-DeepLab-L), a backbone learning rate multiplier of 0.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “• When there is a single client and Θ = Θ1:K , then LocalUpdate recovers the Lookahead optimizer (Zhang et al., 2019) with K “fast weights”.”
Adaptive Federated Optimization (2020, background): “…learning community because they can reduce the need to tune the learning rates and often exhibit strong empirical performance especially in presence of heavy-tail noise distributions of stochastic gradients (Reddi et al., 2019; McMahan and Streeter, 2010; Duchi et al., 2011; Zhang et al., 2019).”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “For training, we use the Ranger optimizer, a combination of Rectified Adam [30] with the Lookahead technique [45], with a constant learning rate of 0.”
Robust fine-tuning of zero-shot models (2021, background): “[108] propose optimizing with a set of fast and slow weights, where every k steps, these two sets of weights are averaged and a new trajectory begins.”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, methodology): “For the question answering experiment we down-loaded 12 an already ﬁne-tuned model from the Hug-12 Tieleman and Hinton (2012); Zhang et al. (2019). gingface implementation 13 of Wolf et al. (2019) We report hyperparameters used by them for training the original model and the ones used for our D IFF…”
Meta-Learning Extractors for Music Source Separation (2020, methodology): “We train using RAdam [7] with the Lookahead optimizer [8] for a max of 250 iterations.”
NAMSG: An Efficient Method For Training Neural Networks (2019, methodology): “h attention. RADAM improves the effectiveness and robustness of ADAM by rectifying the variance of the adaptive learning rate, and RANGER further accelerates RADAM by combining lookahead optimization [35]. In order to further accelerate the convergence of ﬁrstorder methods, in this paper we propose ”
Personalized Federated Learning using Hypernetworks (2021, neutral): “This update rule is inspired by Zhang et al. (2019).”
Learning to Predict Gradients for Semi-Supervised Continual Learning (2022, methodology): “Moreover, Zhang et al . propose an optimization method that wraps an arbitrary optimization method as a component to improve the learning stability [54].”
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models (2019, methodology): “We choose to use the recently introduced Lookahead [9] optimizer as we find that it reduces the number of different seeds we have to try for a successful inversion.”