Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization

By Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, Bohyung Han. 2017

233 citations recorded by the source corpus.

Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Towards Robust Neural Networks via Random Self-ensemble (2017, methodology): “t in a vector. The reason that our RSE algorithm achieves the similar prediction accuracy with original network is because (5) is minimizing an upper bound of the loss of (6) { Similar to the idea of [22], if we choose negative log-likelihood loss, then 8w2Rd w: 1 jD trainj X (x i;y i)2D train E ˘N(”
Universal Multimodal Representation for Language Understanding (2023, background): “In this scenario, the visual features might only provide the regularization effect to improve the model robustness [80], [81], [82].”
Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey (2020, background): “Addition of noise to the image can help the model to learn properly [19].”
Appendix for: How Good is the Bayes Posterior in Deep Neural Networks Really? (2020, unknown): “The connection between dropout and applying Jensen’s bound has been discovered before by several groups (Noh et al., 2017), (Nalisnick et al., 2019), (Inoue, 2019), and contrasts sharply with the variational inference interpretation of dropout, (Kingma et al., 2015; Gal & Ghahramani, 2016).”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…Jiang et al. (2018) and Song et al. (2018) veriﬁed their model on WebVision. et al., 2017) and other regularization techniques (Azadi et al., 2016; Noh et al., 2017), label cleaning/correction (Reed et al., 2014; Goldberger & Ben-Reuven, 2017; Li et al., 2017b; Veit et al., 2017), example…”
Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences (2018, background): “We believe that the accuracy gain is partly by virtue of the stochastic regularization with multiple samples [19].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…learning handle noisy data in multiple directions including dropout (Arpit et al., 2017) and other regularization techniques (Azadi et al., 2016; Noh et al., 2017), label cleaning/correction (Reed et al., 2014; Goldberger & Ben-Reuven, 2017; Li et al., 2017b; Veit et al., 2017; Song et al.,…”