Progressive Blockwise Knowledge Distillation for Neural Network Acceleration

By Hui Wang, Hanbin Zhao, Xi Li, Xu Tan. 2018

73 citations recorded by the source corpus.

As an important and challenging problem in machine learning and computer vision, neural network acceleration essentially aims to enhance the computational efficiency without sacrificing the model accuracy too much. In this paper, we propose a progressive blockwise learning scheme for teacher-student model distillation at the subnetwork block level. The proposed scheme is able to distill the knowledge of the entire teacher network by locally extracting the knowledge of each block in terms of progressive blockwise function approximation. Furthermore, we propose a structure design criterion for the student subnetwork block, which is able to effectively preserve the original receptive field from the teacher network. Experimental results demonstrate the effectiveness of the proposed scheme against the state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Following (Hinton, Vinyals, and Dean 2015), researchers pay more attention to the supervision from intermediate representations for better optimization performance (Romero et al. 2014; Wang et al. 2018; Shen et al. 2019a,b; Ye et al. 2020; Luo et al. 2020).”
Distilling Knowledge From a Deep Pose Regressor Network (2019, methodology): “KD methods for classification [16, 25, 20, 36, 32, 19, 23] rely solely on the teacher prediction without considering the error made w.”
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation (2019, methodology): “To address this problem, the second category of knowledge distillation proposes to employ the internal representation of the teacher to guide the training of the student [26, 37, 36, 33, 25].”
LIT: Learned Intermediate Representation Training for Model Compression (2019, unknown): “Wang et al. (2018) extend FitNets by training networks iteratively using hints.”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Following [8], some works are proposed to exploit the intermediate representation to optimize the learning of student network, such as FitNet [16], DK2PNet [27], AT [33] and NST [26].”
Residual Knowledge Distillation (2020, methodology): “…tell whether the outputs of S and T are close enough, Ashok et al. [1] exploited reinforcement learning to ﬁnd out the best network structure of S under the guidance of T , and Wang et al. [19], Gao et al. [7] referred to the idea of progressive learning to make knowledge transferred step by step.”