Knowledge Distillation by On-the-Fly Native Ensemble

By Xu Lan, Xiatian Zhu, Shaogang Gong. 2018

547 citations recorded by the source corpus.

Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultaneously establishing a strong teacher on-the- fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Moreover, all teachers and students in distillation share the same dataset [38, 40, 41, 42], while in our setting each member (client) can only access its own independent dataset.”
Contrastive Representation Distillation (2019, methodology): “Results on ImageNet For a fair comparison with Zagoruyko & Komodakis (2016a) and Lan et al. (2018), we adopt the models from these papers, ResNet-34 as the teacher and ResNet-18 as the student.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, methodology): “Also, studies such as DML (Zhang et al. 2017) and ONE (Lan, Zhu, and Gong 2018) use the same kind of network for on-line training of peer networks with mutual KLdivergence losses.”
Similarity-Preserving Knowledge Distillation (2019, background): “[18] proposed the onthe-fly native ensemble teacher model, in which the teacher is trained together with multiple students in a multi-branch network architecture.”
Knowledge Distillation: A Survey (2020, background): “…network is usually chosen to be: (1) a simpliﬁed version of a teacher network with fewer layers and fewer channels in each layer (Wang et al. 2018a; Zhu and Gong 2018; Li et al. 2020d); or (2) a quantized version of a teacher network in which the structure of the network is pre-served (Polino et…”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Generalization Analysis Previous works (Yang et al., 2018; Lan et al., 2018) have shown that knowledge distillation can help a model generalize well to unseen data, and thus yield better performance.”
Online Knowledge Distillation with Diverse Peers (2019, methodology): “Together with Table 2 and Figure 4, we can ﬁnd that the compared method CL-ILR and ONE improve the accuracy of individual student models ranging from 0.7% to 8% while decreasing the diversity among different peers, which sacriﬁces the effectiveness of ensemble models.”
On the Efficacy of Knowledge Distillation (2019, methodology): “But does knowledge distillation fulfill this promise of generality? Unfortunately, in spite of the recent interest in variants of knowledge distillation [17, 24, 23, 21, 27, 26, 5, 14], an empirical answer to this question is missing.”
Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “to various domains and have demonstrated promising results [7, 30, 47, 48, 55].”
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (2020, methodology): “Although KD based methods were primarily proposed for model compression [14,37,51,28,53,21,45,26], there have been many attempts to extend them to other tasks recently.”
Adaptive Regularization of Labels (2019, methodology): “Not coincidentally, previous researches [7, 20, 22, 31, 30, 15, 17, 1] have suggested that the cross entropy loss based on one-hot encoded labels may not be optimal in the classification task.”
Hydra: Preserving Ensemble Diversity for Model Distillation (2020, methodology): “By multiplying the loss term by a factor of T 2 we ensure that the relative contributions to additional regularization losses remain roughly unchanged (Song & Chai, 2018; Lan et al., 2018).”
Feature-Level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks (2020, background): “On the contrary, the ones which use an ensemble of networks can transfer ensemble knowledge and also have advantages of the peerteaching framework [7, 11, 15].”
Self-Referenced Deep Learning (2018, background): “Knowledge distillation has been exploited to distil easy-to-train large networks into harderto-train small networks [36], or transfer high-level semantics to earlier layers [26], or simultaneously enhance and transfer knowledge on-the-fly [25].”
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks (2020, unknown): “Some works also proposed online distillation [51, 60] that do not rely on a pre-trained teacher, so teacher and student can be learned simultaneously.”
Knowledge Distillation Meets Self-Supervision (2020, background): “SP [43], Online-KD [24], KD [16], AT [46], and CRD [42].”
QKD: Quantization-aware Knowledge Distillation (2019, methodology): “In general, it has been shown that a powerful teacher with high accuracy can teach a student better than a weaker one [54].”