Automated Inference with Adaptive Batches

By Soham De, Abhay Kumar Yadav, David W. Jacobs, Tom Goldstein. 2017

85 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “ICLR 2018 We note that a number of recent works have discussed increasing the batch size during training (Friedlander & Schmidt, 2012; Byrd et al., 2012; Balles et al., 2016; Bottou et al., 2016; De et al., 2017), but to our knowledge no paper has shown empirically that increasing the batch size”
Visualizing the Loss Landscape of Neural Nets (2017, background): “[23] arguing that generalization is not directly related to the curvature of loss surfaces, and some authors proposing specialized training methods that achieve good performance with large batch sizes [19, 14, 6].”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “A number of authors have proposed adjusting the batch size adaptively during training (Friedlander & Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al.”
Variance-based Gradient Compression for Efficient Distributed Deep Learning (2018, background): “(De et al., 2017) A sufﬁcient condition that a vector - g is a descent direction is (2) We are interested in the case of g = ∇ f B ( x ) , the gradient vector of the loss function over B .”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, background): “Further, there are also important motivations to study the behavior of these algorithms in the deterministic setting because of usecases where the amount of noise is controlled during optimization, either by using larger batches [27, 8, 3] or by employing variance-reducing techniques [17, 9].”
Big Batch SGD: Automated Inference using Adaptive Batch Sizes (2016, background): “d search over dierent possible parameter values. A preliminary version of this paper appears in AISTATS 2017 (International Conference on Articial Intelligence and Statistics), Fort Lauderdale, USA De et al. (2017). This is the extended version. 1 arXiv:1610.05792v4 [cs.LG] 6 Apr 2017 In this paper,”
On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization (2019, background): “…the worst case, e.g., stochastic optimization (1) or large-scale optimization with limited budgets of SFO access, SGD with large batch sizes considered in (De et al., 2017) can have worse convergence performance than SGD with fixed small batch sizes (Bottou & Bousquet, 2008; Bottou et al., 2018).”
Efficient Distributed SGD with Variance Reduction (2015, background): “One approach to control the gradient noise is to grow the batch size over time [8, 10, 14].”
A Progressive Batching L-BFGS Method for Machine Learning (2018, methodology): “Progressive batching (sometimes referred to as dynamic sampling) has been well studied in the optimization literature, both for stochastic gradient and subsampled Newtontype methods (Byrd et al., 2012; Friedlander & Schmidt, 2012; Cartis & Scheinberg, 2015; Pasupathy et al., 2015; Roosta-Khorasani &”
On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “Note that, with a very carefully tuned learning rate schedule, many batch sizes might exhibit both the curvature dominated regime (typically early in training) and the noise dominated regime (late in training) (Sutskever et al., 2013; De et al., 2017; Zhang et al., 2019).”