Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification

By Chong Wang, Xipeng Lan, Yang Zhang. 2017

26 citations recorded by the source corpus.

Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Tao Mei
6. Byeongho Heo
7. Zhen Lei
8. Sangdoo Yun

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xiaogang Wang
3. Luc Van Gool
4. Yi Yang
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Bernt Schiele
8. Andrew Zisserman

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “In addition, the KD-based face recognition can be extended to face alignment and verification by changing the losses in knowledge distillation (Wang et al. 2017).”
• Knowledge Distillation with Adversarial Samples Supporting Decision Boundary (2018, background): “Some studies [4, 19, 5] extended knowledge distillation to computer vision applications.”
• Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “Wang et al. [40] use both the soften and one-hot probability consistency knowledge for face recognition and alignment.”