Adaptive Regularization of Labels

By Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, Shutao Xia. 2019

33 citations recorded by the source corpus.

Recently, a variety of regularization techniques have been widely applied in deep neural networks, such as dropout, batch normalization, data augmentation, and so on. These methods mainly focus on the regularization of weight parameters to prevent overfitting effectively. In addition, label regularization techniques such as label smoothing and label disturbance have also been proposed with the motivation of adding a stochastic perturbation to labels. In this paper, we propose a novel adaptive label regularization method, which enables the neural network to learn from the erroneous experience and update the optimal label representation online. On the other hand, compared with knowledge distillation, which learns the correlation of categories using teacher network, our proposed method requires only a minuscule increase in parameters without cumbersome teacher network. Furthermore, we evaluate our method on CIFAR-10/CIFAR-100/ImageNet datasets for image recognition tasks and AGNews/Yahoo/Yelp-Full datasets for text classification tasks. The empirical results show significant improvement under all experimental settings.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jianping Gou
2. Dacheng Tao
3. Baosheng Yu
4. S. Maybank
5. Lin Wang
6. Kuk-Jin Yoon
7. Changxing Ding
8. Chao-qun Hong

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “…the response-based knowledge has a similar motivation to label smoothing and the model regularization (Kim and Kim 2017; Muller et al. 2019; Ding et al. 2019); The featured-based knowledge is often used to mimic the intermediate process of the teacher and the relation-based knowledge is…”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[13] instead propose residual label and residual loss to enable the student to use the erroneous experience during the training phase, preventing over-fitting and improving the performance.”