Online Hyper-Parameter Learning for Auto-Augmentation Strategy

By Chen Lin, Minghao Guo, Chuming Li, Wei Wu, Dahua Lin, Wanli Ouyang, Junjie Yan. 2019

93 citations recorded by the source corpus.

Data augmentation is critical to the success of modern deep learning techniques. In this paper, we propose Online Hyper-parameter Learning for Auto-Augmentation (OHL-Auto-Aug), an economical solution that learns the augmentation policy distribution along with network training. Unlike previous methods on auto-augmentation that search augmentation strategies in an offline manner, our method formulates the augmentation policy as a parameterized probability distribution, thus allowing its parameters to be optimized jointly with network parameters. Our proposed OHL-Auto-Aug eliminates the need of re-training and dramatically reduces the cost of the overall search process, while establishes significantly accuracy improvements over baseline models. On both CIFAR-10 and ImageNet, our method achieves remarkable on search accuracy, 60x faster on CIFAR-10 and 24x faster on ImageNet, while maintaining competitive accuracies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qi Tian
2. Xinggang Wang
3. Lingxi Xie
4. Xin Chen
5. Qian Zhang
6. Xiaopeng Zhang
7. Wenyu Liu
8. Zhaobai Zhong

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Qi Tian
3. Lingxi Xie
4. Wenyu Liu
5. Qian Zhang
6. Yi Yang
7. Alan L. Yuille
8. Dacheng Tao

Citations in the corpus

• DADA: Differentiable Automatic Data Augmentation (2020, neutral): “Again, OHL AA [16] uses a larger and dynamics search space, thus, it is not comparable to other methods.”
• Adversarial AutoAugment (2019, background): “Hence, several works (Lemley et al., 2017; Cubuk et al., 2018; Lin et al., 2019; Ho et al., 2019) have attempted to automatically learn data augmentation policies.”
• ResizeMix: Mixing Data with Preserved Object Information and True Labels (2020, methodology): “When equipped with RandAugment [9] and the batch augmentation strategy [28,34,54] (the enlarging scale is set as 2 in our experiments), ResizeMix+ achieves top-1 accuracy rates of 98.”
• Image Data Augmentation for Deep Learning: A Survey (2022, methodology): “Unlike auto-augmentation methods searching strategies in an offline manner, [ Lin et al. , 2019 ] formulates the augmentation policy as a parameterized probability distribution and the parameters can be optimized jointly with network parameters, known as OHL-Auto-Aug.”
• Improving One-Shot NAS by Suppressing the Posterior Fading (2019, unknown): “In recent years, there has been a growing interest in developing algorithmic NAS solutions to automate the manual process of architecture design [39, 16, 18].”
• Circumventing Outliers of AutoAugment with Knowledge Distillation (2020, methodology): “ough bayesian optimization, and population-based augmentation [17] applied evolutionary algorithms to generate policy schedule by only a single run of 16 child models. Online hyper-parameter learning [26] combined the search stage and the network training process, and faster AutoAugment [13] formula”
• TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation (2021, unknown): “Automatic augmentation methods have flourished especially for image classification in recent years [2, 13, 14, 9] with many different approaches that learn policies over aug-1 mentation combinations.”