Shake-Shake regularization of 3-branch residual networks

By Xavier Gastaldi. 2017

69 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets (2017, background): “…a lower resolution of the images would make the classiﬁcation task much more difﬁcult and would thus postpone the saturation of benchmarking results currently observed on CIFAR-10, e.g., 3% error obtained by Gastaldi (2017) compared to roughly 6% obtained by a trained human (Karpathy, 2011).”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, methodology): “We tested in the context of image classification that the hinge loss performs as well as the cross entropy on a state-of-the-art architecture [64]: we ran the implementation [65] for CIFAR-10 and we retrained it by replacing the cross entropy by the hinge loss.”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “A small version of the state-of-the-art Shake-Shake model [10] with 2.9M parameters achieves 96.45% accuracy, while our 3.3M parameters model achieves a 96.59% accuracy.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “When trained with a strong regularization technique, such as Shake-Shake (Gastaldi, 2016), and a data augmentation technique, such as CutOut ( DeVries & Taylor, 2017), DenseNet impressively achieves the test error of 2 .”
DADA: Differentiable Automatic Data Augmentation (2020, neutral): “rmance with the full SVHN training data. Unlike AA, we use the Wide-ResNet-28-10 [29] architecture in the search stage and evaluate the policy on the Wide-ResNet-28-10 [29] and Shake-Shake (26 2x96d) [6]. Our results are shown in Table 3. As shown in Table 3, our DADA achieves similar error rate to ”
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (2019, methodology): “ataset Model Baseline Cutout [6] AA [5] PBA [12] Fast AA [18] Faster AA (ours) CIFAR-10 WideResNet-40-2 [35] 5.3 4.1 3.7 - 3.6 3.7 WideResNet-28-10 [35] 3.9 3.1 2.6 2.6 2.7 2.6 Shake-Shake (26 2 32d) [8] 3.6 3.0 2.5 2.5 2.7 2.7 Shake-Shake (26 2 96d) [8] 2.9 2.6 1.9 2.0 2.0 2.0 Shake-Shake (26 2 112”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “For all tests other than the initial SMASHv1 experiments, we employ a cosine annealing schedule [22] without restarts [11].”
Shakedrop Regularization for Deep Residual Learning (2018, methodology): “Recently, an effective regularization method which achieved the lowest test error called Shake-Shake regularization [4], [5] was proposed.”
FreezeOut: Accelerate Training by Progressively Freezing Layers (2017, methodology): “FreezeOut employs cosine annealing (as proposed by [7]) without restarts (as used by [2]) with a layer-wise schedule, where the first layer’s learning rate is reduced to zero partway through training (at t0), and each subsequent layer’s learning rate is annealed to zero some set time thereafter.”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, methodology): “As a benchmark, we retrained the implementation in [17] by replacing the cross entropy by the hinge loss (adapted for multiple classes) without any other modifications.”
Normalized Direction-preserving Adam (2017, methodology): “In addition, for clearer comparisons, the learning rate is annealed according to a cosine function without restart (Loshchilov & Hutter, 2016; Gastaldi, 2017).”
Regularizing Neural Networks via Adversarial Model Perturbation (2020, background): “Shake-Shake regularization [11] combines parallel branches with a stochastic affine function in multibranch networks.”
Macro Neural Architecture Search Revisited (2018, unknown): “DenseNet-BC (k=40) (Gao Huang, 2017) 25.6 17.18 Shake-Shake (Gastaldi, 2017) 26.2 15.85 SMASHv2 (Brock et al., The importance of initial model.”