Online Knowledge Distillation with Diverse Peers
By Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, Chun Chen. 2019
364 citations recorded by the source corpus.
Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this work, we propose Online Knowledge Distillation with Diverse peers (OKDDip), which performs two-level distillation during training with multiple auxiliary peers and one group leader. In the first-level distillation, each auxiliary peer holds an individual set of aggregation weights generated with an attention-based mechanism to derive its own targets from predictions of other auxiliary peers. Learning from distinct target distributions helps to boost peer diversity for effectiveness of group-based distillation. The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i.e., the model used for inference. Experimental results show that the proposed framework consistently gives better performance than state-of-the-art approaches without sacrificing training or inference complexity, demonstrating the effectiveness of the proposed two-level distillation framework.
Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Science Applications
In Computer Vision and Pattern Recognition
In Efficiency & Compression
In Optimization & Training
Citations in the corpus
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
- Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, background): “Knowledge distillation [2,4] is an effective way to transfer knowledge between models.”
- Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “For the two image classification tasks CIFAR-10 and CIFAR-100, our implementation is based on the source code released by Chen et al. (2020)4.”
- Knowledge Distillation: A Survey (2020, background): “A variety of online knowledge distillation methods have been proposed, especially in the last few years (Zhang et al. 2018b; Chen et al. 2020a; Xie et al. 2019; Anil et al. 2018; Kim et al. 2019b; Zhou et al. 2018; Walawalkar et al. 2020; Wu and Gong 2021; Zhang et al. 2021a).”
- MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, background): “Online KD [54,57,3,41,40,45,39] aims to train multiple student networks and perform knowledge transfer within the cohort in an online manner.”
- Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, neutral): “In this case, the diversity of soft labels should be large enough to obtain the effect of the ensemble well (Chen et al. 2020).”
- FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning (2020, background): “Several recent works (Anil et al., 2018; Guo et al., 2020; Chen et al., 2020) investigated the use of model ensemble and knowledge distillation (Hinton et al., 2015) in an online fashion to jointly learn multiple models, where the base models and distillation have access to the centralized labeled…”
- Curriculum Temperature for Knowledge Distillation (2022, background): “Most existing works (Tung and Mori 2019; Chen et al. 2020; Ji et al. 2021) ignore the flexible role of the temperature and empirically set it to a fixed value (e.”
- Safe Distillation Box (2021, background): “Many recent efforts have focused on improving the efficiency of KD, and showcased that the KD process can be lightened without much compensation on performances (Park et al. 2019; Mirzadeh et al. 2020; Chen et al. 2020).”
- Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, background): “Some work such as OKDDip [2] also attempts to leverage self-attention mechanism to compute weights in online KD manner.”
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Whereas, Roheda et al. [183], Zhai et al. [262], Li et al. [126], Chen et al. [30] and Liu et al. [140] focus on discriminating the feature space of teacher and student under CGAN framework.”
- Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [25, 33, 36, 32, 28, 12].”
- Cross-Layer Distillation with Semantic Calibration (2020, background): “To save the expense of pre-training, some cost-effective online variants have been explored later (Anil et al. 2018; Chen et al. 2020).”