Shakedrop Regularization for Deep Residual Learning

By Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, Koichi Kise. 2018

179 citations recorded by the source corpus.

Overfitting is a crucial problem in deep neural networks, even in the latest network architectures. In this paper, to relieve the overfitting effect of ResNet and its improvements (i.e., Wide ResNet, PyramidNet, and ResNeXt), we propose a new regularization method called ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake, which is an effective regularization method, but can be applied to ResNeXt only. ShakeDrop is more effective than Shake-Shake and can be applied not only to ResNeXt but also ResNet, Wide ResNet, and PyramidNet. An important key is to achieve stability of training. Because effective regularization often causes unstable training, we introduce a training stabilizer, which is an unusual use of an existing regularizer. Through experiments under various conditions, we demonstrate the conditions under which ShakeDrop works well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Furthermore, with a more advanced architecture, PyramidNet+ShakeDrop, UDA achieves a new state-of-the-art error rate of 2.7.”
Attention Augmented Convolutional Networks (2019, methodology): “We use standard CIFAR preprocessing: mean normalizing, random flipping and cropping [52, 10, 46].”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “We recognize this as a severe conceptual limitation as CNNs are generally data hungry [26].”
DADA: Differentiable Automatic Data Augmentation (2020, neutral): “Following [3,10,15], we search the DA policy using a Wide-ResNet40-2 network [29] and evaluate the searched policy using Wide-ResNet-40-2 [29], Wide-ResNet-28-10 [29], Shake-Shake [6] and PyramidNet+ShakeDrop [27].”
Fast AutoAugment (2019, methodology): “We examine Wide-ResNet-40-2, Wide-ResNet-28-10 [40], Shake-Shake [8], Shake-Drop [37] models to evaluate the test set accuracy of FAA.”
Meta Pseudo Labels (2020, methodology): “3% [76] but this accuracy is achieved with a PyramidNet which has 17x more parameters than our WideResNet-28-2 and uses the complex ShakeDrop regularization [80].”
Neural networks with late-phase weights (2020, methodology): “, 2017) with ShakeDrop regularization (Yamada et al., 2019).”
On Feature Normalization and Data Augmentation (2020, methodology): “…/ 4.07 StochDepth (Huang et al., 2016) 15.86 / 3.33 DropBlock (Ghiasi et al., 2018) 15.73 / 3.26 Mixup (Zhang et al., 2018) 15.63 / 3.99 ShakeDrop (Yamada et al., 2018) 15 quire more training updates to converge with data augmentation, we apply MoEx to ResNet-50, ResNeXt-50 (32 × 4d),…”
Delving Deep Into Label Smoothing (2020, unknown): “Recently, researchers have proposed many regularization approaches, including Label Smoothing [10], Bootstrap [11], CutOut [12], MixUp [13], DropBlock [14] and ShakeDrop [15], to conquer the over-ﬁtting problem to the distribution of the training set.”