The general inefficiency of batch training for gradient descent learning

By Dennis G. Wilson, Tony R. Martinez. 2003

479 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “Wilson and Martinez (2003) investigated the relationship between batch size and training speed for plain mini-batch SGD.”
FaceNet: A unified embedding for face recognition and clustering (2015, methodology): “On the one hand we would like to use small mini-batches as these tend to improve convergence during Stochastic Gradient Descent (SGD) [20].”
Revisiting Small Batch Training for Deep Neural Networks (2018, background): “…that the discussion about the scaling of the learning rate with the batch size depends on how the problem is formalized, and revert to the view of Wilson & Martinez (2003), that considers the SGD weight update formulation based on the sum, instead of the average, of the gradients over the…”
Deep Speaker: an End-to-End Neural Speaker Embedding System (2017, methodology): “In both stages, we use synchronous SGD with 0.99 momentum [24], with a linear decreasing learning rate schedule from 0.05 to 0.005.”
Systematic evaluation of CNN advances on the ImageNet (2016, background): “Batch size and learning rate The mini-batch size is always a trade-off between computation efficiency – because GPU architecture prefers it large enough – and accuracy; early work by Wilson and Martinez [48] shows superiority of the online training to batchtraining.”
Reinforcement Learning in Continuous State and Action Spaces (2012, background): “There is some indication that often stochastic gradient descent converges faster than batch gradient descent (Wilson and Martinez, 2003).”
Deep learning in robotics: a review of recent research (2017, methodology): “neural networks is stochastic gradient descent [37], but improved methods such as RMSProp [38] and Adam [39] have recently garnered widespread usage.”
The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs (2016, methodology): “In practice, one can use stochastic gradient descent (SGD) [52] to update the parameters {Wl} l=0 .”
Working hard to know your neighbor's margins: Local descriptor learning loss (2017, background): “It is known that small mini-batches are beneﬁcial to faster convergence and better generalization [32].”
Active Sampler: Light-weight Accelerator for Complex Data Analytics at Scale (2015, background): “For similar reasons, batch gradient methods [29] are very expensive for large training datasets.”
Learning Scalable Deep Kernels with Recurrent Structure (2016, background): “However, stochastic gradient methods have been proved to attain better generalization (Hardt et al., 2016) and often demonstrate superior performance in deep and recurrent architectures (Wilson and Martinez, 2003).”
Semantic Annotation of High-Resolution Satellite Images via Weakly Supervised Learning (2016, methodology): “To solve the optimization problem in (2), we use a minibatch gradient descent scheme [44] to obtain the parameters{W1,W2, b1,b2}.”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
Systematic evaluation of convolution neural network advances on the Imagenet (2017, background): “Batch size and learning rate The mini-batch size is always a trade-off between computation efficiency – because GPU architecture prefers it large enough – and accuracy; early work by Wilson and Martinez [48] shows superiority of the online training to batchtraining.”