Regularization for Deep Learning: A Taxonomy

By Jan Kukacka, Vladimir Golkov, Daniel Cremers. 2017

371 citations recorded by the source corpus.

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, background): “on the training data one of the most popular ways to regularize is regularization via data [2].”
Inductive biases for deep learning of higher-level cognition (2020, background): “There are many ways to encode such biases—e.g. explicit regularisation objectives (Bishop et al., 1995; Bishop, 1995; Srivastava et al., 2014; Kukačka et al., 2017; Zhang et al., 2017), architectural constraints (Yu and Koltun, 2015; Long et al., 2015; Dumoulin and Visin, 2016; He et al., 2016;…”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, background): “There are dozens of regularization techniques to combine in deep learning, and each may have multiple hyperparameters [12].”
Fully-Connected Neural Networks with Reduced Parameterization for Predicting Histological Types of Lung Cancer from Somatic Mutations (2020, methodology): “Another approach is to use regularization methods—including dropout, early stopping, and weight decay—that imposes some penalty on the model to reduce its test error but not its training error [5].”
Thin MobileNet: An Enhanced MobileNet Architecture (2019, background): “B. Modiﬁcation 2 - MobileNet architecture with Drop-Activation layers instead of ReLU : Regularization [22, 31, 32] has been an important part of Deep learning networks.”
Deep learning in bioinformatics: introduction, application, and perspective in big data era (2019, methodology): “A systematic review of regularization methods used to combat the overfitting issue in deep learning can be referred to [74].”
3D Deep Learning on Medical Images: A Review (2020, methodology): “This scenario leads to overfitting and there have been several strategies [26] to tackle the problem, such as early stopping of the training epochs and weight penalties (L1 and L2 regularizations, soft weight sharing, and pooling).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “This is because performance differences can often be attributed to factors unrelated to visual input, such as regularization (Kukačka et al., 2017), data bias (Jabri et al.”
Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part III (2021, background): “Regularization is one of the crucial elements of deep learning to prevent the model from overfitting the training data and increase its ability to generalize well on unseen, possibly out-of-distribution test data [27].”
Instance Adaptive Self-Training for Unsupervised Domain Adaptation (2020, background): “Regularization: Regularization refers to schemes that are intended to reduce the testing error and thus make the trained model generalize well to unseen data [7,15].”
A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay (2018, background): “The interested reader can see Kukačka et al. (2017) for a review of regularization methods.”
A review of machine learning applications in wildfire science and management (2020, methodology): “However, 1246 several recent advances have been made to reduce overfitting in ML models, for instance, regularization 1247 techniques in DNNs [Kukačka et al., 2017]. Moreover, when interpreting comparisons between ML and 1248 statistical methods, we should be cognizant that just as some ML methods r”
Text Data Augmentation for Deep Learning (2021, methodology): “Other regularization techniques have been developed such as dropout [4] or weight penalties [5].”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “LeNet5 predates both the current Deep Learning revolution and the so-called AI Winter, dating back to the late 1990s [79].”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, unknown): “In addition to DA, there are some other data manipulation methods to improve model generalization [116, 92].”