Online Batch Selection for Faster Training of Neural Networks

By Ilya Loshchilov, Frank Hutter. 2015

351 citations recorded by the source corpus.

Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset. While it is commonly accepted that batch size is an important parameter for offline tuning, the benefits of online selection of batches remain poorly understood. We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam. As the loss function to be minimized for the whole dataset is an aggregation of loss functions of individual datapoints, intuitively, datapoints with the greatest loss should be considered (selected in a batch) more frequently. However, the limitations of this intuition and the proper control of the selection pressure over time are open questions. We propose a simple strategy where all datapoints are ranked w.r.t. their latest known loss value and the probability to be selected decays exponentially as a function of rank. Our experimental results on the MNIST dataset suggest that selecting batches speeds up both AdaDelta and Adam by a factor of about 5.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Training Region-Based Object Detectors with Online Hard Example Mining (2016, background): “Complementary to [22], we focus on online hard example selection strategy for region-based object detectors.”
Distributed Prioritized Experience Replay (2018, background): “An alternative is to rank samples according to their latest known loss value and make the sampling probability a function of the rank rather than of the loss itself (Loshchilov & Hutter, 2015).”
Recent advances in convolutional neural networks (2015, background): “Online bootstrapping (or hard negative mining [139]) for CNN training has recently gained interest due to its importance for intelligent cognitive systems interacting with dynamically changing environments [58], [140], [141].”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “Selecting more difficult instances can accelerate learning or improve testing accuracy [18, 29, 13].”
Meta-Learning in Neural Networks: A Survey (2020, background): “tion probability [145] or neural network that picks instances [146] for inclusion in a minibatch.”
Bridging Category-level and Instance-level Semantic Image Segmentation (2016, methodology): “Loshchilov and Hutter [20] studied mini-batch selection in terms of image classiﬁcation.”
Supervision via competition: Robot adversaries for learning tasks (2016, methodology): “Recent methods in computer vision [14]–[17] and sample efficiency in reinforcement learning [18] apply hard example mining”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, neutral): “18 However, they use these measures to reweight all instances, similar to sampling effective batches in online learning (Loshchilov and Hutter, 2016).”
Recurrent Pixel Embedding for Instance Grouping (2017, background): “While other work has advocated online bootstrapping methods for hard-pixel mining or mini-batch selection [61, 47, 79, 89], our approach is much simpler.”
CMA-ES for Hyperparameter Optimization of Deep Neural Networks (2016, neutral): “We considered both the default (shufﬂing) and online loss-based batch selection of training examples (Loshchilov & Hutter, 2015).”
High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks (2016, methodology): “Loshchilov and Hutter [15] studied mini-batch selection in terms of image classiﬁcation.”
Not All Samples Are Created Equal: Deep Learning with Importance Sampling (2018, methodology): “For the method of Schaul et al. (2015), we use their proportional sampling since the rank based is very similar to Loshchilov & Hutter (2015) and we select the best parameters from the grid a = { 0 .”
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection (2017, background): “Other examples of findind hard examples for training include [20, 35, 41].”
FairBatch: Batch Selection for Model Fairness (2020, methodology): “More recently, importance sampling techniques have been proposed for faster convergence (Loshchilov and Hutter, 2016; Alain et al., 2016; Stich et al., 2017; Csiba and Richt ´ arik, 2018; Katharopoulos and Fleuret, 2018; Johnson and Guestrin, 2018).”
Learning What Data to Learn (2017, background): “The latter one is limited and time-consuming, as show by an example that the complicated rules in (Loshchilov & Hutter, 2016) accelerate MNIST training, but fail on Cifar10.”
Data Selection for Language Models via Importance Resampling (2023, positive): “Following RoBERTa [48],”
VERI-Wild: A Large Dataset and a New Method for Vehicle Re-Identification in the Wild (2019, background): “[11] proposed to online select hard examples according to loss in SGD optimization.”
Unsupervised Hard Example Mining from Videos for Improved Object Detection (2018, background): “Hard negative mining has also improved the performance of deep learning based models [47,35,19,46,64,55,33].”