Augment Your Batch: Improving Generalization Through Instance Repetition

By Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, Daniel Soudry. 2020

256 citations recorded by the source corpus.

Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling for a fixed budget of optimization steps. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Training data-efficient image transformers & distillation through attention (2020, methodology): “We also use repeated augmentation [4, 23], which provides a significant boost in performance.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We apply AutoAugment (Krizhevsky et al., 2012; Sato et al., 2015; Cubuk et al., 2019; Hoffer et al., 2020) for image augmentation, following the SOTA vision recognition methods (Touvron et al., 2021; Xie et al., 2020b).”
Pix2seq: A Language Modeling Framework for Object Detection (2021, methodology): “For each image in a mini-batch, we perform two independent augmentations, similar to (Hoffer et al., 2020), resulting in a 256 effective batch size, which we ﬁnd helpful to reduce overﬁtting.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “We include most of the augmentation and regularization strategies of [57] in training, except for repeated augmentation [28] and EMA [41], which do not enhance performance.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, background): “DeiT (Touvron et al., 2020) that builds on ViT experimented with various augmentation techniques (Zhang et al., 2017; Yun et al., 2019; Berman et al., 2019; Hoffer et al., 2020; Cubuk et al., 2020), and found them beneﬁcial for ViT training.”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “For data augmentation, extensive tricks are put to use, such as mixup [19], cutmix [18], Rand-Augment [2], repeated augmentation [1, 5], and random cropping.”
Going deeper with Image Transformers (2021, methodology): “The main parameters are as follows: we use a batch size of 1024 samples and train during 400 epochs with repeated augmentation [5, 29].”
Multiscale Vision Transformers (2021, methodology): “We train for 100 epochs with 3 repeated augmentation [50] repetitions (overall computation equals 300 epochs), using a batch size of 4096 in 64 GPUs.”
Transformer in Transformer (2021, background): “The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, methodology): “We also apply drop path [34] for model regularization but instance repetition [16] is only enabled for CrossViT-18 as it does not improve small models.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “In DeiT [3], the need for the large pre-training dataset is replaced with a student-teacher setup and stronger data augmentation and regularization, such as stochastic depth [27] or repeated augmentation [28, 29].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “augmentation [4, 35] and exponential moving average (EMA) [49].”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “Task dataset input Epochs batch learning LR warmup weight repeated size size rate scheduler epoch decay aug [15]”
Global Filter Networks for Image Classification (2021, background): “repeated augmentation [17], which are important to train DeiT while sightly hurting the performance of our models.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, unknown): “, 2017a) 120 epochs 800 / 1600 / 3200 repeated sampling (Hoffer et al., 2020) 4 augmentation hflip, crop [0.”
Co-Scale Conv-Attentional Image Transformers (2021, methodology): “For fair comparison, we perform data augmentation such as MixUp [42], CutMix [41], random erasing [45], repeated augmentation [12], and label smoothing [28], following identical procedures in DeiT [30].”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “We use the same set of data augmentation and regularization strategies used in [55] after excluding random erasing [82], repeated augmentation [4, 34] and exponential moving average (EMA) [48].”