ShakeDrop regularization

By Yoshihiro Yamada, Masakazu Iwamura, Koichi Kise. 2018

71 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “The Shake-Shake model [12] employed a 26 2 × 96d conﬁgura-tion, and the PyramidNet model used the ShakeDrop regularization [51].”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…evaluate SAM’s impact on generalization for today’s state-of-the-art models on CIFAR-10 and CIFAR-100 (without pretraining): WideResNets with ShakeShake regularization (Zagoruyko & Komodakis, 2016; Gastaldi, 2017) and PyramidNet with ShakeDrop regularization (Han et al., 2016; Yamada et al., 2018).”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “As can be seen from the table, we achieve an error rate of 1.48% with the ShakeDrop [60] model, which is 0.65% better than the state-of-the-art [21].”
Born Again Neural Networks (2018, methodology): “It is only beaten by Yamada et al. (2018) who use a pyramidal ResNet trained for 1800 epochs with a combination of shake-shake (Gastaldi, 2017), pyramid-drop (Yamada et al., 2016) and cut-out regularization (DeVries & Taylor, 2017).”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, methodology): “The models include widely used convolutional networks (VGG and ResNet [7, 18]), more recent architectures (ResNeXt, PyramidNet, DenseNet [6, 10, 20]), the published state-of-the-art (Shake-Drop [21]), and a model derived from RL-based hyperparameter search (NASNet) [23].”
Adversarial AutoAugment (2019, methodology): “We evaluate our proposed method with the following models: Wide-ResNet-28-10 (Zagoruyko & Komodakis, 2016), Shake-Shake (26 2x32d) (Gastaldi, 2017), Shake-Shake (26 2x96d) (Gastaldi, 2017), Shake-Shake (26 2x112d) (Gastaldi, 2017), PyramidNet+ShakeDrop (Han et al., 2017; Yamada et al., 2018).”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, neutral): “(Gastaldi, 2017), Shake-Shake (26 2x96d) (Gastaldi, 2017), Shake-Shake (26 2x112d) (Gastaldi, 2017), and Pyramid-Net with ShakeDrop (Han et al., 2016; Yamada et al., 2018).”
Learning data augmentation policies using augmented random search (2018, methodology): “Additionally, we achieve an error rate of 1.26 % with the ShakeDrop [19] model, which is 0.22 % better than AutoAug-ment.”
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, neutral): “…(Huang et al., 2017) 25.6M 3.46 PyramidNet (Han et al., 2017) 26.0M 3.31 Shake-Shake + c/o (DeVries & Taylor, 2017) 26.2M 2.56 PyramidNet + SD (Yamada et al., 2018) 26.0M 2.31 ENAS + c/o (Pham et al., 2018) 4.6M 2.89 DARTS + c/o (Liu et al., 2018c) 3.4M 2.83 NASNet-A + c/o (Zoph et al., 2018)…”
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, methodology): “…network using the regularization methods of CutOut (DeVries and Taylor, 2017), MixUp (Hongyi Zhang, 2018), Shake-Shake (Gastaldi, 2017), and Shake-Drop (Yamada et al., 2018) 1 , as well as the optimization algorithm of stochastic gradient descent with restarts (SGDR; Loshchilov and Hutter (2017)).”
DropBlock: A regularization method for convolutional networks (2018, methodology): “Since its introduction, dropout [1] has inspired a number of regularization methods for neural networks such as DropConnect [14], maxout [15], StochasticDepth [16], DropPath [17], Sched-uledDropPath [8], shake-shake regularization [18], and ShakeDrop regularization [19].”
Shakedrop Regularization for Deep Residual Learning (2018, background): “This paper is an extended version of ICLR workshop paper [35].”
NSML: Meet the MLaaS platform with a real-world case study (2018, background): “(Yamada et al., 2018; Xiong et al., 2018; Liang et al., 2018; Yu et al., 2018; Min et al., 2017) As the applications of the deep learning grow, the complexity of the tasks for designing and tuning the neural network architectures increases as well as the size of models.”
AutoAugment: Learning Augmentation Strategies From Data (2019, methodology): “Our best model, Pyramid-Net+ShakeDrop trained with AutoAugment achieves an error rate of 4.4% (2.9% higher than the error rate on the original set).”
On Feature Normalization and Data Augmentation (2020, methodology): “…/ 4.07 StochDepth (Huang et al., 2016) 15.86 / 3.33 DropBlock (Ghiasi et al., 2018) 15.73 / 3.26 Mixup (Zhang et al., 2018) 15.63 / 3.99 ShakeDrop (Yamada et al., 2018) 15 quire more training updates to converge with data augmentation, we apply MoEx to ResNet-50, ResNeXt-50 (32 × 4d),…”