Shake-Shake regularization

By Xavier Gastaldi. 2017

398 citations recorded by the source corpus.

The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem. The idea is to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination. Applied to 3-branch residual networks, shake-shake regularization improves on the best single shot published results on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%. Experiments on architectures without skip connections or Batch Normalization show encouraging results and open the door to a large set of applications. Code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “We also test whether UDA can benefit from more advanced architectures by using Shake-Shake (26 2x96d) [16] and PyramidNet+ShakeDrop [63] instead of Wide-ResNet-28-2.”
Attention Augmented Convolutional Networks (2019, methodology): “We use standard CIFAR preprocessing: mean normalizing, random flipping and cropping [52, 10, 46].”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “The Shake-Shake model [12] employed a 26 2 × 96d conﬁgura-tion, and the PyramidNet model used the ShakeDrop regularization [51].”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…evaluate SAM’s impact on generalization for today’s state-of-the-art models on CIFAR-10 and CIFAR-100 (without pretraining): WideResNets with ShakeShake regularization (Zagoruyko & Komodakis, 2016; Gastaldi, 2017) and PyramidNet with ShakeDrop regularization (Han et al., 2016; Yamada et al., 2018).”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, neutral): “sophisticated “shake-shake” regularization [15].”
Improved Regularization of Convolutional Neural Networks with Cutout (2017, methodology): “For our tests, we use the original code and training settings provided by the author of [4], with the only change being the addition of cutout.”
Fixing Weight Decay Regularization in Adam (2017, methodology): “Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFAR-10 and CIFAR-100 Krizhevsky (2009), are still obtained by applying SGD with momentum (Huang et al., 2016; 2017; Loshchilov & Hutter, 2016; Gastaldi, 2017).”
Squeeze-and-Excitation Networks (2017, methodology): “Shake-Even 29 2x4x64d [70] + Cutout [71] 15.”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “Baseline Cutout [25] AutoAugment Wide-ResNet-28-10 [57] 18.84 16.50 14.13 Shake-Shake (26 2x96d) [59] 17.05 13.40 10.04 Table 3: Test set error rates (”
Born Again Neural Networks (2018, methodology): “It is only beaten by Yamada et al. (2018) who use a pyramidal ResNet trained for 1800 epochs with a combination of shake-shake (Gastaldi, 2017), pyramid-drop (Yamada et al., 2016) and cut-out regularization (DeVries & Taylor, 2017).”
Neural Architecture Optimization (2018, neutral): “lts show that our NAONet derived with CIFAR-10 is indeed transferable to more complicated task such as CIFAR-100. Model B N F #op Error (%) #params DenseNet-BC [20] / 100 40 / 17.18 25.6M Shake-shake [15] / / / / 15.85 34.4M Shake-shake + Cutout [11] / / / / 15.20 34.4M NASNet-A [51] 5 6 32 13 19.70”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, background): “09% (which is better than the error Gastaldi (2017) obtained with a slightly larger network).”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “We do not use stochastic regularization techniques, such as ShakeShake (Gastaldi, 2017) or ScheduledDropPath (Zoph et al., 2018) in this exper-iment as they are not applicable to all networks out of the box.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, methodology): “Additionally, we experiment with the recent Shake-Shake2x64d [Gastaldi, 2017] on CIFAR-10 and PyramidNet272 (bottleneck, α = 200) [Han et al.”
UniformAugment: A Search-free Probabilistic Data Augmentation Approach (2020, methodology): “The models for the datasets are Shake-Shake 26 2x96d (SS) [6], Wide-ResNet-28-2 (WRN), and ResNet50 (RN50) [7], respectively.”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “In the ﬁnal-retrain phase, the optimal network architecture will be trained with additional regularization techniques, e.g., Shake-Shake (Gastaldi, 2017), DropPath (Larsson et al., 2017), and Cutout (DeVries and Taylor, 2017).”
Adversarial AutoAugment (2019, neutral): “We evaluate our proposed method with the following models: Wide-ResNet-28-10 (Zagoruyko & Komodakis, 2016), Shake-Shake (26 2x32d) (Gastaldi, 2017), Shake-Shake (26 2x96d) (Gastaldi, 2017), Shake-Shake (26 2x112d) (Gastaldi, 2017), PyramidNet+ShakeDrop (Han et al., 2017; Yamada et al., 2018).”