Understanding Short-Horizon Bias in Stochastic Meta-Optimization

By Yuhuai Wu, Mengye Ren, Renjie Liao, Roger Baker Grosse. 2018

146 citations recorded by the source corpus.

Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if meta-optimization is to scale to practical neural net training regimes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, methodology): “Local properties of the loss landscape are not necessarily a good guide to overall optimal training [WRLG18].”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Therefore, we generally tried to select the best validation error that we could achieve reliably within a practical training time. the best schedule is an optimization problem in its own right (Wu et al., 2018).”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “3.1 Problem Setup We now introduce the noisy quadratic model [Schaul et al., 2013, Martens, 2014, Wu et al., 2018], where the true function being optimized is a convex quadratic.”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “iple tasks. Meta-learning methods typically learn the update policy yet lack an overall learning objective in the few-shot episodes. Furthermore, they could potentially suffer from shorthorizon bias (Wu et al., 2018), if at test time the computation graph is unrolled for longer steps. To address thi”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “While simple, this model is a proxy for neural network optimization and effectively optimizing it remains a challenging open problem [37, 26, 43, 47].”
Meta-Learning in Neural Networks: A Survey (2020, background): “memory and compute but suffer from a short-horizon bias [299] and thus have lower meta-test accuracy.”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “Our method improves the training objective through a weighted loss rather than an average loss and is an in-stantiation of meta-learning (Thrun & Pratt, 1998; Lake et al., 2017; Andrychowicz et al., 2016), i.e. learning to learn better.”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, background): “In terms of learning rate decay schedule, [Wu et al., 2018] studied the learning rate schedule via short-horizon bias.”
Meta-Learning with Warped Gradient Descent (2019, methodology): “Unfortunately, because they backpropagate through the adaptation process, we cannot scale them to general forms of meta-learning (Wu et al., 2018; Flennerhag et al., 2019).”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “Optimization-based meta-learning methods typically rely on truncated optimization for the adaptation process [7], to satisfy both computational and memory constraints [36, 28], and MACAW also uses a truncated optimization.”
Gradients are Not All You Need (2021, background): “Depending on the underlying task this bias can be severe [Wu et al., 2016], while for other tasks, empirically, this matters less.”
Transferring Knowledge across Learning Processes (2018, background): “MAML exhibits a considerable short-horizon bias (Wu et al., 2018).”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, background): “Unrolling a small number of steps can be crucial for performance but may induce bias [37].”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “Short-horizon bias phenomena of optimized learning rates were studied in Wu et al. (2018) for the convex quadratic and their theoretical insights were successfully translated to neural networks.”
A Generalizable Approach to Learning Optimizers (2021, methodology): “These methods have the advantage of not requiring training a policy ahead of time, but generally suffer from short-horizon bias [48].”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, background): “Intuitively, this short-horizon (greedy) (Wu et al., 2018) behavior makes sense: the quickest way to make short-term progress (improve L T∗ (θt+1)) is to descend solely on T ∗.”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “, 2018), the bias induced by restricting the number of adaptation steps (Wu et al., 2018), or the effect of higher-order terms in the meta-gradient estimation (Foerster et al.”
Neural networks with late-phase weights (2020, background): “Despite its apparent simplicity, it remains a challenging optimization problem that has important similarities with stochastic minibatch training in deep neural networks (Schaul et al., 2013; Martens, 2016; Wu et al., 2018; Zhang et al., 2019a;b).”