Channel-wise Knowledge Distillation for Dense Prediction*

By Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, Chunhua Shen. IEEE International Conference on Computer Vision, 2020

462 citations recorded by the source corpus.

Knowledge distillation (KD) has been proven a simple and effective tool for training compact dense prediction models. Lightweight student networks are trained by extra supervision transferred from large teacher networks. Most previous KD variants for dense prediction tasks align the activation maps from the student and teacher network in the spatial domain, typically by normalizing the activation values on each spatial location and minimizing point-wise and/or pair-wise discrepancy. Different from the previous methods, here we propose to normalize the activation map of each channel to obtain a soft probability map. By simply minimizing the Kullback–Leibler (KL) divergence between the channel-wise probability map of the two networks, the distillation process pays more attention to the most salient regions of each channel, which are valuable for dense prediction tasks.We conduct experiments on a few dense prediction tasks, including semantic segmentation and object detection. Experiments demonstrate that our proposed method outperforms state-of-the-art distillation methods considerably, and can require less computational cost during training. In particular, we improve the RetinaNet detector (ResNet50 backbone) by 3.4% in mAP on the COCO dataset, and PSPNet (ResNet18 backbone) by 5.81% in mIoU on the Cityscapes dataset. Code is available at: https://git.io/Distiller

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/237c2d518b79678f7332a4243f57a285711a4c37

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS (2023, methodology): “A quantization scheme for detection using RepOptimizer[104] and channel-wise distillation [105] that helped to achieve a faster detector.”
A Comprehensive Review of YOLO: From YOLOv1 to YOLOv8 and Beyond (2023, methodology): “A quantization scheme for detection using RepOptimizer[92] and channel-wise distillation [93] that helped to achieve a faster detector.”
Knowledge Distillation from A Stronger Teacher (2022, methodology): “Following [37, 43, 46], we train DeepLabV3 [6] and PSPNet [54] with ResNet-18 backbone on Cityscapes dataset, and adopt our DIST on the predictions of classification head using a teacher with ResNet101 backbone of DeepLabV3.”
Depth Anything V2 (2024, unknown): “We aim to reveal the importance of large-scale unlabeled data and larger teacher model, rather than delicate loss designs [43, 69] or distillation pipelines [10].”
YOLO-based Object Detection Models: A Review and its Applications (2024, unknown): “Using channel-wise distillation [86] and RepOptimizer [87] a quantization method is developed for achieving a fast detection.”
Effective Whole-body Pose Estimation with Two-stages Distillation (2023, methodology): “From the logit-based distillation to feature-based distillation, the knowledge is transferred from intermediate layers [17, 55, 57] and it extends the distillation to various tasks, including detection [4, 56], segmentation [38], generation [30] and so on.”
YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications (2022, methodology): “Besides, channelwise distillation [36] (later as CW Distill) is adapted within the YOLOv6 framework, shown in Fig.”