Data augmentation instead of explicit regularization

By Alex Hernandez-Garcia, Peter König. 2018

166 citations recorded by the source corpus.

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, background): “Most strategies either add slightly modified copies of existing data or create synthetic data, aiming for the augmented data to act as a regularizer and reduce overfitting when training ML models (Shorten and Khoshgoftaar, 2019; Hernández-García and König, 2020).”
Fast AutoAugment (2019, methodology): “Since extensive data augmentation protects the network from overfitting [14], we believe the performance will be improved by reducing the weight decay which is tuned for the model with default augmentation rules.”
A Group-Theoretic Framework for Data Augmentation (2019, background): “For example, a line of work proposes to add random or adversarial noise when training [11, 51, 26, 74, 8, 49, 73, 37] and argue that this can lead to a form of regularization [63, 34, 35, 18, 71].”
A Survey on Data Augmentation for Text Classification (2021, unknown): “According to Hernandez-Garcia and König [20], data augmentation is a preferred regularization method, as it achieves generalization without degrading the models’ representational capacity and without re-tuning other hyperparameters.”
Tradeoffs in Data Augmentation: An Empirical Study (2021, methodology): “With this deﬁnition, it is widely acknowledged that commonly-used augmentations act as regular-izers (Hernández-García & König, 2018a;b; Zhang et al., 2016; Dao et al., 2019).”
Machine learning in energy economics and finance: A review (2019, methodology): “Both control the effective capacity of the network with the purpose of reducing the overfitting (Hernández-García and König, 2018): - Implicit Regularization: These regularization methods use characteristics of the net- work architecture, the learning algorithm or the data in order to control the…”