DropBlock: A regularization method for convolutional networks

By Golnaz Ghiasi, Tsung-Yi Lin, Quoc V. Le. 2018

1,053 citations recorded by the source corpus.

Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout. Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropbBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices. Extensive experiments show that DropBlock works better than dropout in regularizing convolutional networks. On ImageNet classification, ResNet-50 architecture with DropBlock achieves $78.13\%$ accuracy, which is more than $1.6\%$ improvement on the baseline. On COCO detection, DropBlock improves Average Precision of RetinaNet from $36.8\%$ to $38.4\%$.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

Citations in the corpus

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “Dropblock [8], the feature-level generalization of Cutout, is also more effective when label smoothing is also used.”
Meta-Learning With Differentiable Convex Optimization (2019, methodology): “We use DropBlock regularization [9], a form of structured Dropout.”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “Each MaxPooling layer has a stride of two and is followed by a DropBlock [11] layer with keep prob = 0.8 (block size = 7 for the first one and block size = 5 for the second one).”
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection (2019, methodology): “Following the technique in [9], we apply DropBlock with block size 3x3 after batch normalization layers in the the NAS-FPN layers.”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “Our best model, Auto-DeepLab-L, with single scale inference significantly outperforms [19] by 20.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, methodology): “Implementation details For the FSL experiments on mini-ImageNet and tiered-ImageNet we employ the same ResNet12 with DropBlock [13] as backbone and pretrain it on base classes for 500 epochs with SGD optimizer with momentum with the learning rate (lr) of 1e-3 that is decayed by 0.”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “We use DropBlock regularization [9], a form of structured dropout with a keep-rate of 0.”
Rethinking Pre-training and Self-training (2020, methodology): “[7] find model trained with random initialization outperforms the ImageNet pre-trained model on COCO object detection when strong regularization is applied.”
ResNeSt: Split-Attention Networks (2020, background): “As a structured variant of dropout, DropBlock [18] randomly masks out local block regions, and is more eﬀective than dropout for speciﬁcally regularizing convolutional layers.”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “While for the benchmark experiments on MiniImageNet and TieredImageNet, we follow Lee et al. (2019) to set the architecture of ResNet, which contains 12 layers and uses the DropBlock (Ghiasi et al. 2018) to prevent over-fitting.”
Learning Data Augmentation Strategies for Object Detection (2019, methodology): “Briefly, we trained each child model3 from scratch on the 5K COCO images with the ResNet-50 backbone [16] and RetinaNet detector [24] using a cosine learning rate decay [30].”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “Previous work indicates that improvements on the clean accuracy appear after training with Dropblock for 270 epochs [17], but we did not ﬁnd that training for 270 epochs changed our analysis.”
YOLOv4: Optimal Speed and Accuracy of Object Detection (2020, methodology): “We have verified a large number of BoF, including grid sensitivity elimination, mosaic data augmentation, IoU threshold, genetic algorithm, class label smoothing, cross mini-batch normalization, selfadversarial training, cosine annealing scheduler, dynamic mini-batch size, DropBlock, Optimized Ancho”
Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions (2018, methodology): “4 The DropBlock [3] is used in this ResNet architecture to avoid over-fitting.”
No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, methodology): “…training techniques, which include data augmentation (Cubuk et al., 2018; 2020), regularization (Srivastava et al., 2014; Szegedy et al., 2016; Ghiasi et al., 2018), architectures (Tan & Le, 2019; Dosovitskiy et al., 2020; Hu et al., 2018; Iandola et al., 2014; Li et al., 2019; Szegedy et…”
Revisiting the Sibling Head in Object Detector (2020, background): “b&w indicates training with bells and whistles such as multi-scale train/test, Cascade R-CNN or DropBlock [7].”