Self-supervised Knowledge Distillation Using Singular Value Decomposition

By Seunghyun Lee, Dae Ha Kim, Byung Cheol Song. 2018

153 citations recorded by the source corpus.

To solve deep neural network (DNN)’s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79%. code is available on https://github.com/sseung0703/SSKD_SVD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Class Gradient Projection For Continual Learning (2022, methodology): “Therefore, if we directly perform Singular Value Decomposition (SVD) [21] on the representations from all classes of a task, some bases for “apple” or “car” may be missed.”
Knowledge Distillation: A Survey (2020, methodology): “Using the correlations between feature maps as the distilled knowledge, knowledge distillation via singular value decomposition was proposed to extract key information in the feature maps (Lee et al. 2018).”
Unpaired Multi-Modal Segmentation via Knowledge Distillation (2020, background): “Previous work has adopted knowledge distillation to address various tasks not limited to original model compression [22], but also a wider scope of scenarios such as domain adaptation [23], life-long learning [24], adversarial attacks [25] and selfsupervised learning [26].”
Task-Oriented Feature Distillation (2020, methodology): “AT [70] Channelwise Pooling Channel dims FSP [68] FSP Matrix Spatial dims Jacobian [58] Gradients Channel dims SVD [26] SVD Decomposition Spatial dims Heo et al. [20] Margin ReLU Negative Feature Task-Oriented Convolutional Layers Non-Task-Oriented Feature Distillation Trained by Task Loss features…”
Interactive Knowledge Distillation (2020, methodology): “[12] used Singular Value Decomposition (SVD) to perform dimension reduction on feature maps.”
CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis (2020, methodology): “Lee et al. [38] propose a knowledge distillation method using singular value decomposition to improve the quality of the transferred knowledge.”
Residual Knowledge Distillation (2020, methodology): “Zagoruyko and Komodakis [22] averaged the feature map across channel dimension to obtain spatial attention map, Yim et al. [21] deﬁned inter-layer ﬂow by computing the inner product of two feature maps, and Lee et al. [14] improved this idea with singular value decomposition (SVD).”
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (2020, methodology): “Since then, numerous KD variants [37,51,28,53,21,42,45] have been proposed, mostly focusing on using either feature maps or attention maps at the intermediate layers of the teacher network as the extra hints for improving KD designs.”
Distilling Knowledge From a Deep Pose Regressor Network (2019, methodology): “KD methods for classification [16, 25, 20, 36, 32, 19, 23] rely solely on the teacher prediction without considering the error made w.”
R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning (2022, methodology): “Since then, various KD methods [22,13,26,18] have been developed.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Heterogeneous graph Layer-wise feature L 2 GSP [217] IRG [206] Model compression Knowledge graph Middle layers L 2 Instance relations MHKD [205] Model compression Knowledge graph Middle layers KL SVD [70] + Attention KTG [211] Model compression Directed graph Network model L 1 + KP loss – GFL [212] ”
Knowledge distillation via adaptive instance normalization (2020, background): “The limitation of this work is the high computational cost, which is addressed to some extent by [24] by compressing the feature maps by singular value decomposition.”
Graph-based Knowledge Distillation by Multi-head Attention Network (2019, methodology): “proposed a method to find the relation of feature maps more effectively by using SVD [16].”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, unknown): “[15] proposed to distill the knowledge from an ensemble of models into a single model, there are a lot of follow-up works, including exploring different forms of knowledge [49, 25], cross-modality distillation [13, 1], cross-task distillation [34, 63].”