Augment your batch: better training with larger batches

By Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, Daniel Soudry. 2019

79 citations recorded by the source corpus.

Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, neutral): “We compare this recipe to the one used for DeiT models in the Appendix, and observe that our setup provides substantially faster training convergence likely because we remove repeating augmentation [1, 20], which is known to slow training [1].”
UniformAugment: A Search-free Probabilistic Data Augmentation Approach (2020, background): “It would be interesting to also deploy UA in a similar setting using batch augmentation[9], to investigate its eﬀectiveness with larger augmented batches.”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “• In-Batch Augmentations: Since activation gradients are not normally aggregated across batch, we use inbatch augmentations [10] technique to gain a batching effect for gradients per samples; Each sample is duplicated N = 4 times with a set of random differential augmentations chosen from a set cont”
ResizeMix: Mixing Data with Preserved Object Information and True Labels (2020, methodology): “When equipped with RandAugment [9] and the batch augmentation strategy [28,34,54] (the enlarging scale is set as 2 in our experiments), ResizeMix+ achieves top-1 accuracy rates of 98.”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, background): “In non-private training, several authors have shown that using multiple augmentations per unique example in a mini-batch can improve performance (Fort et al., 2021; Hoffer et al., 2019; Touvron et al., 2021b).”
Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks (2022, background): “[22] Batch Augment Increases the size of the mini-batch Input”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, methodology): “As stated above, augmentation multiplicity was ﬁrst proposed by Hoffer et al. [2019] as a promising strategy for large batch training (i.e. by increasing the batch size as the augmentation multiplicity increases), while Berman et al. [2019] found it can also enhance performance when the batch size…”
A Group-Theoretic Framework for Data Augmentation (2019, background): “For example, a line of work proposes to add random or adversarial noise when training [11, 51, 26, 74, 8, 49, 73, 37] and argue that this can lead to a form of regularization [63, 34, 35, 18, 71].”
AutoFormer: Searching Transformers for Visual Recognition (2021, methodology): “Data augmentation techniques, including RandAugment [10], Cutmix [68], Mixup [69] and random erasing, are adopted with the same hyperparameters as in DeiT [50] except the repeated augmentation [20].”