Residual error based knowledge distillation

By Mengya Gao, Yujun Wang, Liang Wan. 2020

47 citations recorded by the source corpus.

Abstract Knowledge distillation (KD) is one of the most popular ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this paper proposes Residual error based Knowledge Distillation(RKD), which further distills the knowledge by introducing an assistant model(A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification datasets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods and keep strong robustness to adversarial samples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition