Deep Networks with Stochastic Depth

By Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q. Weinberger. 2016

2,707 citations recorded by the source corpus.

Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91 % on CIFAR-10).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “…Published 12 LeaderBoard 13 Single Model EM / F1 EM / F1 LR Baseline (Rajpurkar et al., 2016) 40.4 / 51.0 40.4 / 51.0 Dynamic Chunk Reader (Yu et al., 2016) 62.5 / 71.0 62.5 / 71.0 Match-LSTM with Ans-Ptr (Wang & Jiang, 2016) 64.7 / 73.7 64.7 / 73.7 Multi-Perspective Matching (Wang et…”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “[52] 0.1 mixup [69] 0.8 cutmix [68] 1.0 drop path [30] 0.1 (B/L) 0.2 (H) Partial ﬁne-tuning.”
Gaussian Error Linear Units (GELUs) (2016, background): “ng. The stochasticity of dropout and zoneout may allow for a pseudo-ensemble [1] of networks with stochastic width or stochastic depth, respectively, and this leads to marked performance improvements [5, 14]. Since ReLUs lack stochasticity and since the aforementioned regularizers are irrespective o”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “We also use swish activation (Ramachandran et al., 2018; Elfwing et al., 2018), ﬁxed Au-toAugment policy (Cubuk et al., 2019), and stochastic depth (Huang et al., 2016) with drop connect ratio 0.2.”
Wide Residual Networks (2016, background): “ms CIFAR-10 CIFAR-100 NIN [20] 8.81 35.67 DSN [19] 8.22 34.57 FitNet [24] 8.39 35.04 Highway [28] 7.72 32.39 ELU [5] 6.55 24.28 original-ResNet[11] 110 1.7M 6.43 25.16 1202 10.2M 7.93 27.82 stoc-depth[14] 110 1.7M 5.23 24.58 1202 10.2M 4.91 - pre-act-ResNet[13] 110 1.7M 6.37 - 164 1.7M 5.46 24.33 10”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “Additionally, we set the stochastic depth survival probability [37] and dropout ratio to be 0.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “We use stochastic depth [29], dropout [63] and RandAugment [14] to noise the student.”
HyperNetworks (2016, background): “The original resnet and subsequent derivatives (Zhang et al., 2016; Huang et al., 2016a) achieved state-of-the-art image recognition performance on a variety of public datasets.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, methodology): “…state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, Ima-geNet, PASCAL VOC and MS COCO datasets were obtained by Residual Neural Networks (He et al., 2015; Huang et al., 2016c; He et al., 2016; Zagoruyko & Komodakis, 2016) trained without the use of advanced methods such as AdaDelta and Adam.”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, background): “However, we found that drop path regularization [24, 22] is highly effective for ViT back-bones and therefore we include it ( e.g ., it improves from-scratch training by up to 2 AP box ).”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “We employ stochastic depth (Huang et al., 2016) with a 0.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “We have employed stochastic depth [27], which facilitates the convergence”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “The drop rate for Stochastic Depth [17] is set to 0.”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “batch normalization (Ioffe & Szegedy, 2015), stochastic depth (Huang et al., 2016), data augmentation (Cubuk et al.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “, 2014) or stochastic depth (Huang et al., 2016) as well as data augmentation methods such as SpecAugment (Park et al.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “LARGE trains for 250k updates, BASE for 400k updates, and LARGE on LV-60k for 600k updates.”
Segment Anything (2023, methodology): “To regularize SAM, we set weight decay ( wd ) to 0.1 and apply drop path [53] ( dp ) with a rate of 0.4.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “To reduce overfitting for XLS-R (2B), we increase stochastic depth to 15% compared to 10% for all other setups (Huang et al., 2016; Fan et al., 2019).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “This is inspired by the Stochastic Depth approach of Huang et al. (2016) used to train very deep ResNets (He et al., 2015).”