Learning Student Networks via Feature Embedding

By Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, Dacheng Tao. 2018

116 citations recorded by the source corpus.

Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teacher network. Traditional teacher–student-based methods used to rely on additional fully connected layers to bridge intermediate layers of teacher and student networks, which brings in a large number of auxiliary parameters. In contrast, this article aims to propagate information from teacher to student without introducing new variables that need to be optimized. We regard the teacher–student paradigm from a new perspective of feature embedding. By introducing the locality preserving loss, the student network is encouraged to generate the low-dimensional features that could inherit intrinsic properties of their corresponding high-dimensional features from the teacher network. The resulting portable network, thus, can naturally maintain the performance as that of the teacher network. Theoretical analysis is provided to justify the lower computation complexity of the proposed method. Experiments on benchmark data sets and well-trained networks suggest that the proposed algorithm is superior to state-of-the-art teacher–student learning methods in terms of computational and storage complexity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yunhe Wang
2. Dacheng Tao
3. Kai Han
4. Chunjing Xu
5. Jianyuan Guo
6. Jianping Gou
7. S. Maybank
8. Baosheng Yu

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Dacheng Tao
3. Kai Han
4. Jianyuan Guo
5. Chunjing Xu
6. Jianping Gou
7. S. Maybank
8. Xinghao Chen

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “, 2020; Tung and Mori, 2019) The locality similarities of feature representations of hint layers of the teacher networks (Chen et al., 2020); The similar activations of input pairs in the teacher networks (Tung and Mori, 2019).”
• Learning Compatible Embeddings (2021, background): “Various knowledge sources are adopted in literature such as classification logits [17, 1, 28], hint layers [35, 54, 49, 3] and multi-layer groups [52, 51, 20].”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “Chen et al. [4] borrowed the idea of manifold learning to design a novel knowledge distillation method, in which the student preserves the feature embedding similarities of 2 2 ( , ) t t p p samples from the teacher.”
• Distilling Object Detectors via Decoupled Features (2021, methodology): “Since then, KD has been widely adopted in classification tasks [22, 63, 3, 54, 7, 65, 10].”
• Lipschitz Continuity Guided Knowledge Distillation (2021, methodology): “Authors [25, 4, 39] constructed a similarity measurement for feature representations using singular value decomposition (SVD) to elicit relations between different layers as transferred knowledge.”