Coupling Adaptive Batch Sizes with Learning Rates

By Lukas Balles, Javier Romero, Philipp Hennig. 2016

124 citations recorded by the source corpus.

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates. This variance also changes over the optimization process; when using a constant batch size, stability and convergence is thus often enforced by means of a (manually tuned) decreasing learning rate schedule. We propose a practical method for dynamic batch size adaptation. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance proportionally to the value of the objective function, removing the need for the aforementioned learning rate decrease. In contrast to recent related work, our algorithm couples the batch size to the learning rate, directly reflecting the known relationship between the two. On popular image classification benchmarks, our batch size adaptation yields faster optimization convergence, while simultaneously simplifying learning rate tuning. A TensorFlow implementation is available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “In a classic paper, Bottou and Bousquet [2008] studied the asymptotics of stochastic optimization algorithms and found SGD to be competitive with fancier approaches. They showed that stochastic optimization involves fundamentally different tradeoffs from full-batch optimization. More recently, sever”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “We note that a number of recent works have discussed increasing the batch size during training (Friedlander & Schmidt, 2012; Byrd et al., 2012; Balles et al., 2016; Bottou et al., 2016; De et al., 2017), but to our knowledge no paper has shown empirically that increasing the batch size and decaying…”
Early Stopping without a Validation Set (2017, background): “4, 5 and 7, this results in an early stopping criterion for stochastic gradient descent: Remark on implementation: Computing the stopping criterion is straight-forward, given that the variance estimate ˆΣ is available.”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “Then, by using the approximation η l L l ≈ 1 , which is common in SGD literature (Balles et al., 2016), we derive the following adaptive strategy: Apart from coupling the communication period with learning rate, when to decay the learning rate is another key design factor.”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “, 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework.”
Stochastic natural gradient descent draws posterior samples in function space (2018, result): “…at constant temperature is largely independent of the learning rate, indicating that stochastic NGD obeys a linear scaling rule between batch size and learning rate, B ∝ (cid:15) when B N , as already observed by many authors for SGD (Goyal et al., 2017; Smith et al., 2018; Balles et al., 2016).”
BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy (2019, background): “However, it is known that a large mini-batch beyond a certain point can degrade training quality (Balles et al., 2016; Keskar et al., 2016; Krizhevsky, 2014), not to mention that mini-batch size is limited by the GPU memory capacity in practice.”
Follow the Signs for Robust Stochastic Optimization (2017, background): “Several recent papers [11, 2, 12] have pointed out that an explicit, local estimate of the gradient variance r(θ) can be obtained from a single mini-batch evaluation via r̂(θ) = 1 |B| − 1 (”
BackPACK: Packing more into backprop (2019, methodology): “An empirical estimate of the (marginal) variance of the gradients within the batch has been found to be theoretically and practically useful for adapting hyperparameters like learning rates (Mahsereci & Hennig, 2017) and batch sizes (Balles et al., 2017), or regularize firstorder optimization (Le Ro”
Optimal SGD Hyperparameters for Fully Connected Networks (2018, background): “As discussed in [1, 2, 9, 5, 11], the continuum limit of stochastic gradient descent is described by the stochastic differential equation dwi = −∂iLτdt+ dξi , E[dξidξj ]|w,t = 2gFij(w)dt , (4) where wi is used to denote the parameters of the neural network, and dξi is random noise.”
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks (2017, background): “In contrast an adaptive criterion to control the batch size increases and illustration of their convergence on convex problems and convolutional neural networks is developed by De et al. (2016) and Balles et al. (2017).”
Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients (2017, background): “The (non-trivial) implementation of this estimator for neural networks is described in Balles et al. (2017a).”