Data-Distortion Guided Self-Distillation for Deep Neural Networks

By Ting-Bing Xu, Chenglin Liu. 2019

207 citations recorded by the source corpus.

Knowledge distillation is an effective technique that has been widely used for transferring knowledge from a network to another network. Despite its effective improvement of network performance, the dependence of accompanying assistive models complicates the training process of single network in the need of large memory and time cost. In this paper, we design a more elegant self-distillation mechanism to transfer knowledge between different distorted versions of same training data without the reliance on accompanying models. Specifically, the potential capacity of single network is excavated by learning consistent global feature distributions and posterior distributions (class probabilities) across these distorted versions of data. Extensive experiments on multiple datasets (i.e., CIFAR-10/100 and ImageNet) demonstrate that the proposed method can effectively improve the generalization performance of various network architectures (such as AlexNet, ResNet, Wide ResNet, and DenseNet), outperform existing distillation methods with little extra training efforts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “For ResNet32, the Snapshot Distillation method of Xu & Liu (2019) outperforms our own, although it does not do so for ResNet110 and is not evaluated for ResNet56.”
Knowledge Distillation: A Survey (2020, methodology): “In self-distillation, the same networks are used for the teacher and the student models (Zhang et al., 2019b; Hou et al., 2019; Yang et al., 2019b; Lee et al., 2019a; Lan et al., 2018; Phuong and Lampert, 2019b; Xu and Liu, 2019).”
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, methodology): “Moreover, there are literatures on a self-knowledge distillation, which trains the student network progressively to distill and to regularize its own knowledge without the pretrained teacher network [43, 35, 18, 32, 39].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “lation. Although most methods focus on how to better supervise student in self-distillation, data representations for training the student are not fully excavated and utilized. To this end, Xu et al. [163] and Lee et al. [165] focus on selfdistillation via data augmentation of the training samples, ”
MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, methodology): “Existing Self-KD works often employ auxiliary architecture [31,52,53,18,51] or data augmentation [37,48] to capture additional knowledge to enhance the network.”
Regularizing Class-Wise Predictions via Self-Knowledge Distillation (2020, methodology): “We compare our method with prior regularization methods such as the state-of-the-art angular-margin based methods [5, 58], entropy regularization [13, 32, 36, 43] and self-distillation methods [53, 57].”
Delving Deep Into Label Smoothing (2020, unknown): “-based self-KD [24] Data-based self-KD [25] models from overﬁtting to some speciﬁc samples, Dubey et al. [27] added pairwise confusion to the output logits of samples belonging to different categories in training so that the models can learn slightly less discriminative features for speciﬁc samples.”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “Thus, the model compression and acceleration techniques mainly including network pruning [2, 3], model quantization [34, 35] and knowledge distillation [20, 21, 39, 40, 36] are proposed for training lightweight deep models.”