Fast AutoAugment

By Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, Sungwoong Kim. 2019

659 citations recorded by the source corpus.

Data augmentation is an essential technique for improving generalization ability of deep learning models. Recently, AutoAugment has been proposed as an algorithm to automatically search for augmentation policies from a dataset and has significantly enhanced performances on many image recognition tasks. However, its search method requires thousands of GPU hours even for a relatively small dataset. In this paper, we propose an algorithm called Fast AutoAugment that finds effective augmentation policies via a more efficient search strategy based on density matching. In comparison to AutoAugment, the proposed algorithm speeds up the search time by orders of magnitude while achieves comparable performances on image recognition tasks with various models and datasets including CIFAR-10, CIFAR-100, SVHN, and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Although this assumption is sufﬁcient for identifying learned augmentation policies to improve performance [5, 57, 36, 25, 20], it is unclear if this assumption is overly stringent and may lead to sub-optimal data augmentation policies.”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, neutral): “We have ensured that our implementations’ generalization performance in the absence of SAM matches or exceeds that reported in prior work (Cubuk et al., 2018; Lim et al., 2019) All results use basic data augmentations (horizontal flip, padding by four pixels, and random crop).”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “of 1:2 10 4. We train the models for 800 epochs using a batch size of 1;024images and using N= 32;000negative samples in Equation3. We do not use data-augmentation approaches such as Fast AutoAugment [38] because they are the result of supervised-learning approaches. We provide a full overview of al”
Prototypical Contrastive Learning of Unsupervised Representations (2020, methodology): “report top-1 accuracy. Numbers with are from released pretrained model; all other numbers are adopted from corresponding papers. y: LocalAgg uses 10-crop evaluation. CMC and ADMIM use FastAutoAugment [44] that is supervised by ImageNet labels. SimCLR requires a large batch size of 4096 allocated on ”
AutoML: A Survey of the State-of-the-Art (2019, background): “[46] propose fast autoaugment (FastAA), which finds more effective augmentation policies via a more efficient search strategy based on density matching.”
Toward Causal Representation Learning (2021, methodology): “…over all distributions in E , one can relax the problem by sampling from the interventional distributions and optimize an expectation over the different augmented images on a suitably chosen subset [39], using a search algorithm, such as RL [49] or an algorithm based on density matching [154].”
UniformAugment: A Search-free Probabilistic Data Augmentation Approach (2020, methodology): “Fast AutoAugment (FAA) [12] utilizes Bayesian Optimization to ﬁnd a data augmentation policy trained for density matching to improve the generalization ability of the model but this again requires an expensive separate search phase on a proxy task, in order of hundreds of GPU hours.”
DADA: Differentiable Automatic Data Augmentation (2020, neutral): “Recently, inspired by the neural architecture search (NAS) algorithms, some methods [3, 10, 15, 16] attempted to automate learning data augmentation policies.”
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (2019, methodology): “tion. One direction is to search for better combinations of symbolic operations using black-box optimization techniques: reinforcement learning [5, 23], evolution strategy [32], Bayesian optimization [18] and Population Based Training [12]. As the objective, [5, 32, 12] directly aim to minimize erro”
Albumentations: fast and flexible image augmentations (2018, methodology): “AutoAugment [16] and Fast AutoAugment [40] used reinforcement learning to find the optimal data augmentation strategies from a discrete search space of transform operations.”
Towards Causal Representation Learning (2021, neutral): “algorithm based on density matching [154].”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, methodology): “We consider various perturbation types including simple single-image-based DA perturbing the original input image, mixed-labeled DA which integrates multiple images [49, 51] and automated DAs (AutoDAs) [9, 11, 32].”
Supervised Contrastive Learning (2020, positive): “We experimented with various data augmentations and found that AutoAugment [29] gave us the best results in terms of downstream performance.”
Adversarial Examples Improve Image Recognition (2019, background): “Recent works also demonstrate that it is possible to learn data augmentation policies automatically for achieving better performance on image classiﬁcation [16, 1, 2, 17] and object detection [36, 2].”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, background): “This process is related to what is called density-matching in Fast AutoAugment (Lim et al., 2019), where policies are optimized so that the density of augmented validation images match the density of images from the training set.”
Learning Data Augmentation Strategies for Object Detection (2019, methodology): “Data augmentation strategies for vision models are often focused on the image classification domain [21, 15, 41, 3, 6, 17, 28].”