Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

By Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, Kaisheng Ma. 2019

1,120 citations recorded by the source corpus.

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation - a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices. Our codes have been released on github.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Large Language Models Can Self-Improve (2022, neutral): “While our setting is self-improvement and different from knowledge distillation to smaller models (Hinton et al., 2015; Zhang et al., 2019), CoT reasoning (Wei et al.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, background): “More recently, other works [38], [39] explore an interesting variation of self-distillation that distills knowledge from the deeper layers of the network to its shallower ones to accelerate convergence and attain higher accuracy.”
Knowledge Distillation: A Survey (2020, methodology): “knowledge and distillation, the relationship between the structures of the teacher and the student also signiﬁcantly inﬂuences the performance of knowledge distillation. For example, as described in (Zhang et al., 2019b), the student model may learn very little from some teacher models. This is caus”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “[44] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.”
Meta Pseudo Labels (2020, background): “These softmax predictions are generally called the soft labels, which have been widely utilized in the literature on knowledge distillation [26, 17, 86].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “The choices of teacher models are also well studied, where [227, 265] use multiple teacher models to jointly supervise the student model, while [43, 269] apply self-distillation without an extra teacher model.”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “This approach is further supported by the ML community’s growing number of works on progressive networks [23, 35, 48, 72, 81, 83, 84] which can be directly targeted by SPINN to yield an optimised deployment on mobile platforms.”
Self-supervised Label Augmentation via Input Transformations (2019, methodology): “They utilize partially-independent architectures (Lan et al., 2018), data distortion (Xu & Liu, 2019), or hidden layers (Zhang et al., 2019b) for distillation.”
Continual Contrastive Self-supervised Learning for Image Classification (2021, background): “[57][58] propose to transfer attention map information to student.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Typical approaches include pruning [12], [13], [14], [15], [16], quantization [17], [18], [19], [20], light-weight neural network design [21], [22], [23], [24], [25], low rank factorization [26], [27] and knowledge distillation [28], [29], [30], [31].”
Training Graph Neural Networks with 1000 Layers (2021, methodology): “RevGAT-SelfKD uses selfknowledge distillation (Zhang et al., 2019) with 5 attention layers, 3 heads, and 256 channels each.”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “Self-distillation is a novel training scheme for knowledge transfer [23, 24, 27, 11, 10].”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “Zhang et al. [221] introduced a self-supervised spectral–spatial attention-based vision transformer (SSVT), where global and local augmented views are contrasted based on self-distillation [222].”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, methodology): “FastBERT, on the other hand, uses self-distillation (Zhang et al., 2019; Phuong and Lampert, 2019) for ﬁne-tuning, which works well for small Muppets such as BERT BASE .”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, positive): “To reduce total training time, online distillation (Yang et al. 2019; Zhang et al. 2019) is proposed to unify the training of student and teacher into one step.”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, background): “Inspired by [13, 51], our aim is not to compress a model yet to improve its performance.”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, background): “Recently, many variants and extensions are proposed [6, 18, 24–26, 35, 37, 44, 47, 49, 54], such as employing internal feature representations [30], adversarial training with discriminators [32], transfer ﬂow [48], contrastive distillation [40], patient and consistent [2] etc.”
Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (2024, methodology): “Compared to traditional self-distillation (SD) [72], RSD enhances the final predictor on purpose.”