Structured Knowledge Distillation for Semantic Segmentation

By Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, Jingdong Wang. 2019

708 citations recorded by the source corpus.

In this paper, we investigate the issue of knowledge distillation for training compact semantic segmentation networks by making use of cumbersome networks. We start from the straightforward scheme, pixel-wise distillation, which applies the distillation scheme originally introduced for image classification and performs knowledge distillation for each pixel separately. We further propose to distill the structured knowledge from cumbersome networks into compact networks, which is motivated by the fact that semantic segmentation is a structured prediction problem. We study two such structured distillation schemes: (i) pair-wise distillation that distills the pairwise similarities, and (ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by extensive experiments on three scene parsing datasets: Cityscapes, Camvid and ADE20K.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Knowledge distillation and its improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda (2021, methodology): “Theses methods explore the image crop/resizing [367] or efficient network design [368], [369] or transfer learning [370], [371].”
Enforcing Geometric Constraints of Virtual Normal for Depth Prediction (2019, methodology): “Various DCNN-based methods focus on designing structural features [33, 48, 13], especially in depth prediction.”
Interactive Knowledge Distillation (2020, background): “To reduce the resource consumption while maintaining good performance, researchers leverage knowledge distillation techniques [6], [7], [8], [9], [10], [11] to transfer informative knowledge from a cumbersome but well-trained teacher network into a light-weight but unskilled student network.”
Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes (2023, methodology): “Some methods utilize knowledge distillation technology to improve the performance while introducing no cost in the inference stage [25], [26], [27].”
Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy (2020, methodology): “Input Teacher MIMIC [22] + CE Pair-wise [28] + CE CSC + CE”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, background): “ing [41, 61], depth estimation [22], action recognition [66], semantic segmentation [44], self-supervised learning [58] or self-training [76].”
FasterSeg: Searching for Faster Real-time Semantic Segmentation (2019, methodology): “Auto-DeepLab (Liu et al., 2019a) ﬁrst introduced network-level search space to optimize resolutions (in addition to cell structure) for segmentation tasks.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, background): “Wang et al. and Liu et al. apply knowledge distillation on objection detection, segmentation, depth prediction and other vision tasks [39], [49].”
Intra-class Feature Variation Distillation for Semantic Segmentation (2020, methodology): “Liu et al. [24] propose structured knowledge distillation (SKD), which also transfers pairwise relations and forces the outputs of the student model to mimic the teacher model from a holistic view via adversarial learning.”
Channel-wise Knowledge Distillation for Dense Prediction* (2020, methodology): “The pairwise affinity [24, 13, 23] is employed to transfer the similarity between pixel pairs.”
Task-Oriented Feature Distillation (2020, methodology): “Besides image classiﬁcation, knowledge distillation has also been applied in other domains, such as object detection [63], image segmentation [37], nature language processing [8, 59, 28], distributed training [2], semi-supervised learning [32].”
Inter-Region Affinity Distillation for Road Marking Segmentation (2020, neutral): “, with knowledge transferred through softened class scores [6], feature maps matching [9, 13] or spatial attention maps matching [27].”
Adaptive multi-teacher multi-level knowledge distillation (2020, methodology): “Despite image classiﬁcation, the tasks of visual question answering [14], semantic segmentation [12] and machine translation [24] have been also beneﬁted from distillation learning.”
Diverse Knowledge Distillation for End-to-End Person Search (2020, methodology): “It is a common practice to improve the main network using the additional guidance from external networks (Li, Jin, and Yan 2017; Cheng et al. 2018; Zhu et al. 2019; Hinton, Vinyals, and Dean 2015; Tung and Mori 2019; Liu et al. 2019; Yuan et al. 2020).”
Learning Distilled Collaboration Graph for Multi-Agent Perception (2021, methodology): “Knowledge distillation (KD) is a widely used technique to compress a larger teacher network to a smaller student network by pushing the student to approach the teacher in either output or intermediate feature space [10], and it has been applied to various tasks such as semantic segmentation [21], si”
Structured Consistency Loss for semi-supervised semantic segmentation (2020, neutral): “As explained in section 2, we utilize a concept of pair-wise knowledge distillation technique suggested by Liu et al. [13] for the structured consistency loss.”
Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels (2020, neutral): “Current stateof-the-art methods try to further increase the information transfer from the teacher DNN by incorporating various intermediate feature representations and novel loss formulations during training [22], [23], [24], [25], [26].”