DisturbLabel: Regularizing CNN on the Loss Layer

By Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, Qi Tian. 2016

263 citations recorded by the source corpus.

During a long period of time we are combating overfitting in the CNN training process with model regularization, including weight decay, model averaging, data augmentation, etc. In this paper, we present DisturbLabel, an extremely simple algorithm which randomly replaces a part of labels as incorrect values in each iteration. Although it seems weird to intentionally generate incorrect training labels, we show that DisturbLabel prevents the network training from over-fitting by implicitly averaging over exponentially many networks which are trained with different label sets. To the best of our knowledge, DisturbLabel serves as the first work which adds noises on the loss layer. Meanwhile, DisturbLabel cooperates well with Dropout to provide complementary regularization functions. Experiments demonstrate competitive recognition results on several popular image recognition datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Label Refinery: Improving ImageNet Classification through Label Progression (2018, background): “Distur-bLabel [10] replaces some of the labels in a training batch with random labels.”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Several improved smoothing schemes were proposed, including sampling erroneous labels instead of using a fixed distribution [25], using the marginal label probabilities [17], or using early errors of the model [26].”
Object Recognition with and without Objects (2016, methodology): “Recently, several efficient methods were proposed to help CNNs converge faster [16] as well as prevent overfitting [36][12].”
Adaptive Regularization of Labels (2019, methodology): “[26] proposed DisturbLabel method sets randomly wrong labels during each training phase.”
Stochasticity and Skip Connection Improve Knowledge Transfer (2021, background): “This might be problematic since the random noise can corrupt the knowledge in the outputs [14].”
When Does Label Smoothing Help? (2019, background): “butions other than uniform, resulting in unigram label smoothing (see Table 1) which is advantageous when the output labels’ distribution is not balanced. Label smoothing also relates to DisturbLabel [18], which can be seen as label dropout, whereas label smoothing is the marginalized version of lab”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, methodology): “Simply adding label noise has also been shown to be effective at regularizing neural networks (Xie et al., 2016).”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, background): “[41], but setting a minimum k proved more effective.”
SORT: Second-Order Response Transform for Visual Recognition (2017, methodology): “Recently, efficient methods were proposed to help CNNs converge faster and prevent over-fitting, such as ReLU activation [37], Dropout [15], DisturbLabel [53], batch normalization [19] and varying network depth in training [18].”
Does label smoothing mitigate label noise? (2020, background): “On the other hand, smoothing might accentuate the problem, as it is equivalent to injecting uniform noise into all labels [Xie et al., 2016].”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “Lastly, label noise can intentionally be injected in purpose of regularizing [42] or data poisoning [21], [22].”
Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation (2017, methodology): “We emphasize that Noisy Softmax adds noise on a single loss layer instead of on many previous layers, which is more convenient and efﬁcient for implementation and model training, and is applied in DCNNs. Intrinsically different from DisturbLabel [51], where noise is produced by disturbing labels and”
P3SGD: Patient Privacy Preserving SGD for Regularizing Deep CNNs in Pathological Image Classification (2019, background): “adding appropriate randomness into the training phase [39, 42, 45].”
Person Re-identification: Past, Present and Future (2016, background): “This strategy has been evaluated in object classiﬁcation where a small portion of labels are disturbed before training [159].”