Improved Regularization of Convolutional Neural Networks with Cutout

By Terrance Devries, Graham W. Taylor. 2017

4,336 citations recorded by the source corpus.

Convolutional neural networks are capable of learning powerful representational spaces, which are necessary for tackling complex learning tasks. However, due to the model capacity required to capture such representations, they are often susceptible to overfitting and therefore require proper regularization in order to generalize well. In this paper, we show that the simple regularization technique of randomly masking out square regions of input during training, which we call cutout, can be used to improve the robustness and overall performance of convolutional neural networks. Not only is this method extremely easy to implement, but we also demonstrate that it can be used in conjunction with existing forms of data augmentation and other regularizers to further improve model performance. We evaluate this method by applying it to current state-of-the-art architectures on the CIFAR-10, CIFAR-100, and SVHN datasets, yielding new state-of-the-art results of 2.56%, 15.20%, and 1.30% test error respectively. Code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Speciﬁcally, in the fully supervised learning settings, Cubuk et al. [9] also show that AutoAugment improves upon Cutout and that Cutout is more effectively than basic augmentations, which aligns well with the observations in semi-supervised settings.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “This category includes 86 models with an explicit robustness intervention such as adversarially robust models [13, 27, 72, 74, 101], models with special data augmentation [20, 28, 34, 41, 100, 108, 113], and models with architecture modiﬁcations [115].”
Stochastic Optimization of Plain Convolutional Neural Networks with Simple Methods (2020, methodology): “Bengio et al. [5] did apply a large variety of transformations to a handwritten character dataset, for example, local elastic deformation, motion blur, Gaussian smoothing, Gaussian noise, salt and pepper noise, pixel permutation and affine transformations[4].”
Natural Adversarial Examples (2019, background): “Cutout augmentation [11] randomly occludes image regions and corresponds to 4.”
Do ImageNet Classifiers Generalize to ImageNet? (2019, neutral): “wide_resnet_28_10_cutout [11, 61] https://github.”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “Throughout the experiments on ImageNet, we set the default image size to 224 224 except for some specific cases, and adopt some common data augmentation methods such as mixup [76] and cutmix [77], [78] for both ViT, T2T-ViT and VOLO model training, because ViT models need more training data to reach”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “Hyper-parameter settings: We set the hole size of Cutout [3] to 16× 16.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “penalizing the classification network sensitivity to samples generated via MixUp or CutOut [49, 9].”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “For auto-augment, we use the best policy found on this dataset as described in (Cubuk et al., 2018) plus cutout (Devries & Taylor, 2017).”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “Note that, unless otherwise stated, we use Cutout [13] followed by either of these strategies.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “Method 1000 labels 5000 labels CutOut [12] 12.”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, neutral): “5% in the case of STL10 and CI-Cat 74.1 Cutout* 2017 [15] 87.3 DeepCluster 2018 [7] 73.4 ADC 2018 [23] 56.7 IIC plus ﬁnetune 88.8 FAR10.”
A Survey of Data Augmentation Approaches for NLP (2021, neutral): “, 2019), CUTOUT (DeVries and Taylor, 2017) and COPY-PASTE (Ghiasi”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, background): “The other two augmentations, inspired by “Cutout”, proposed in computer vision [19], are time and frequency masking, where we mask a block of consecutive time steps or mel frequency channels.”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “Some works [17,89,19, 23] have proposed augmenting the common datasets (e.”
Designing Network Design Spaces (2020, methodology): “We emphasize that R EG N ET models use our basic 100 epoch schedule with no regularization except weight decay, while most mobile networks use longer schedules with various enhancements, such as deep supervision [16], Cutout [4], DropPath [14], AutoAugment [2], and so on.”
On Network Design Spaces for Visual Recognition (2019, methodology): “As a sanity check, we perform point comparisons using larger models and the exact training settings from DARTS [21] which uses a 600 epoch schedule with deep supervision [18], Cutout [5], and modified DropPath [17].”