Online Knowledge Distillation via Collaborative Learning

By Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, Ping Luo. 2020

374 citations recorded by the source corpus.

This work presents an efficient yet effective online Knowledge Distillation method via Collaborative Learning, termed KDCL, which is able to consistently improve the generalization ability of deep neural networks (DNNs) that have different learning capacities. Unlike existing two-stage knowledge distillation approaches that pre-train a DNN with large capacity as the ''teacher'' and then transfer the teacher's knowledge to another ''student'' DNN unidirectionally (i.e. one-way), KDCL treats all DNNs as ''students'' and collaboratively trains them in a single stage (knowledge is transferred among arbitrary students during collaborative training), enabling parallel computing, fast computations, and appealing generalization ability. Specifically, we carefully design multiple methods to generate soft target as supervisions by effectively ensembling predictions of students and distorting the input images. Extensive experiments show that KDCL consistently improves all the ''students'' on different datasets, including CIFAR-100 and ImageNet. For example, when trained together by using KDCL, ResNet-50 and MobileNetV2 achieve 78.2% and 74.0% top-1 accuracy on ImageNet, outperforming the original results by 1.4% and 2.0% respectively. We also verify that models pre-trained with KDCL transfer well to object detection and semantic segmentation on MS COCO dataset. For instance, the FPN detector is improved by 0.9% mAP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, background): “The multi-branch architecture subsequently became the mainstream for online knowledge distillation (Guo et al., 2020; Chen et al., 2020; Wu and Gong, 2020).”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, methodology): “(1) Logits-based knowledge utilizes the ﬁnal output layer as supervision information to generate student network (Hinton, Vinyals, and Dean 2015; Guo et al. 2020).”
Knowledge Distillation: A Survey (2020, methodology): “2018b), DCM (Yao and Sun 2020) and KDCL (Guo et al. 2020) are the performance of teacher and student after online distillation.”
Improving Plasticity in Online Continual Learning via Collaborative Learning (2023, background): “Collaborative learning [5, 20, 42, 51, 52] orients from online knowledge distillation (KD).”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Collaborative learning is the one used most often [25, 29, 32, 8, 9, 7], where the teacher and the student as peer networks collaboratively teach and learn from each other, and the peer network architectures can be different.”
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning (2020, background): “Several recent works (Anil et al., 2018; Guo et al., 2020; Chen et al., 2020) investigated the use of model ensemble and knowledge distillation (Hinton et al., 2015) in an online fashion to jointly learn multiple models, where the base models and distillation have access to the centralized labeled…”
Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, methodology): “The omni-supervised concept is approached through data distillation [8], which is different to conventional model distillation methods [56], [57] that ensemble multiple experts but entail re-training different cum-bersome networks.”