Sample size selection in optimization methods for machine learning

By Richard H. Byrd, Gillian M. Chin, Jorge Nocedal, Yuchen Wu. 2012

433 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Mathematical Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Mathematical Physics

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “[BCNW12] implicitly uses the gradient noise scale for a theoretical analysis of batch size selection.”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “We note that a number of recent works have discussed increasing the batch size during training (Friedlander & Schmidt, 2012; Byrd et al., 2012; Balles et al., 2016; Bottou et al., 2016; De et al., 2017), but to our knowledge no paper has shown empirically that increasing the batch size and decaying…”
TensorFlow: A system for large-scale machine learning (2016, neutral): “We show in Subsection 6.3 that TensorFlow can train larger models on larger clusters with step times as short as 2 seconds.”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “Besides the works discussed earlier, there have been many recent works analyzing subsampled Newton meth-ods (Byrd et al., 2011, 2012; Milzarek et al., 2019; Li et al., 2019).”
Optimization Methods for Large-Scale Machine Learning (2016, background): “One can, however, employ a backup [28, 70]: if (5.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “ese autoencoder problems, a more slowly increasing schedule, or one that stops increasing well before mreaches jSj, may be more appropriate. One may also consider using an approach similar to that of Byrd et al. (2012) for adaptively determining a suitable mini-batch size. In our second experiment w”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “A number of authors have proposed adjusting the batch size adaptively during training (Friedlander & Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al.”
Coupling Adaptive Batch Sizes with Learning Rates (2016, methodology): “We also compare against a batch size adaptation based on the criterion (13) used in [3] and [6].”
Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales (2019, background): “Such adaptive schemes have been proposed in the literature primarily for gradient descent methods and in a convex setting [5, 13].”
A Stochastic Line Search Method with Convergence Rate Analysis (2018, background): “We observe that: • unlike [5,9], the number of samples for gradient and function estimation does not increase at any pre-defined rate, but is closely related to the progress of the algorithm.”
Extragradient Method with Variance Reduction for Stochastic Variational Inequalities (2017, methodology): “In [9, 5, 13, 38] the SA approach is studied for convex SP.”
Big Batch SGD: Automated Inference using Adaptive Batch Sizes (2016, neutral): “Choose a stepsize α t .”
Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks (2018, methodology): “The general idea of this method [36], [37] is that the initial noisy gradient estimations from small mini-batches explore a relatively flat energy landscape without falling into sharp local minima.”
Sub-Sampled Newton Methods I: Globally Convergent Algorithms (2016, methodology): “In such methods a subset S ⊂ { 1 , 2 , · · · , n } is chosen at random and the update is obtained by”
A Progressive Batching L-BFGS Method for Machine Learning (2018, methodology): “Byrd et al. (2012) proposed a progressive batching strategy, based on a norm test , that determines when to increase the sample size; they established linear convergence and computational complexity bounds in the case when the batch size grows geometrically.”
Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients (2013, background): “While n can be seen as a hyperparameter of the algorithm [9], it is often constrained to a large extent by the computational hardware, memory requirements and communication bandwidth.”