Big Batch SGD: Automated Inference using Adaptive Batch Sizes

By Soham De, Abhay Kumar Yadav, David W. Jacobs, Tom Goldstein. 2016

64 citations recorded by the source corpus.

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We propose alternative "big batch" SGD schemes that adaptively grow the batch size over time to maintain a nearly constant signal-to-noise ratio in the gradient approximation. The resulting methods have similar convergence rates to classical SGD, and do not require convexity of the objective. The high fidelity gradients enable automated learning rate selection and do not require stepsize decay. Big batch methods are thus easily automated and can run with little or no oversight.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “The required geometric growth rate for G k is the same as that of SGD to obtain linear convergence without variance-reduction or interpolation (De et al., 2016; Friedlander and Schmidt, 2012).”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Other authors [12, 17, 22, 42, 62] use a line-search termination criteria that requires function/gradient evaluations averaged over multiple samples.”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “This is a well-known fact in optimization, and most convergence bounds for stochastic-gradient based methods, including minibatch, involve upper bounds on E[‖ĝ(wk)‖](2)2, in a multiplicative form, for both convex and nonconvex setups [39, 40, 41, 42, 42, 43, 44, 45, 46, 47].”
Coupling Adaptive Batch Sizes with Learning Rates (2016, methodology): “[6] increasing the batch size by a small increment whenever the criterion is not satisﬁed, and only then perform the SGD update.”
Training Quantized Nets: A Deeper Understanding (2017, background): “In addition to providing a means of improving fixed-point training, this suggests that recently proposed methods using big batches [28, 29] may be able to exploit lower levels of precision to further accelerate training.”
Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks (2018, unknown): “[40] empirically evaluate the dynamic sampling”
Automated Inference with Adaptive Batches (2017, methodology): “Using this batching strategy, we show that we can keep An extended version of this work is in De et al. (2016). the stepsize constant, or let it adapt using a simple Armijo backtracking line search, making the method completely adaptive with no user-defined parameters.”
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks (2017, background): “In contrast an adaptive criterion to control the batch size increases and illustration of their convergence on convex problems and convolutional neural networks is developed by De et al. (2016) and Balles et al. (2017).”
The Effect of Network Width on the Performance of Large-batch Training (2018, methodology): “Other works propose methods that automatically choose the batch size on the fly [30, 31].”
Gradient Diversity: a Key Ingredient for Scalable Distributed Learning (2017, methodology): “De et al. (2016) propose an optimization algorithm for choosing the batch-size, and weighted sampling techniques have also been developed (Needell and Ward, 2016, Zhang et al., 2017, Zhao and Zhang, 2014).”
Gradient Diversity Empowers Distributed Learning (2017, unknown): “In addition to providing theoretical guarantees for the choice of batch-size, De et al. propose an optimization algorithm for choosing the batch-size [8].”