Face Model Compression by Distilling Knowledge from Neurons

By Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, Xiaoou Tang. 2016

228 citations recorded by the source corpus.

The recent advanced face recognition systems werebuilt on large Deep Neural Networks (DNNs) or theirensembles, which have millions of parameters. However, the expensive computation of DNNs make theirdeployment difficult on mobile and embedded devices. This work addresses model compression for face recognition,where the learned knowledge of a large teachernetwork or its ensemble is utilized as supervisionto train a compact student network. Unlike previousworks that represent the knowledge by the soften labelprobabilities, which are difficult to fit, we represent theknowledge by using the neurons at the higher hiddenlayer, which preserve as much information as the label probabilities, but are more compact. By leveragingthe essential characteristics (domain knowledge) of thelearned face representation, a neuron selection methodis proposed to choose neurons that are most relevant toface recognition. Using the selected neurons as supervisionto mimic the single networks of DeepID2+ andDeepID3, which are the state-of-the-art face recognition systems, a compact student with simple network structure achieves better verification accuracy on LFW than its teachers, respectively. When using an ensemble of DeepID2+ as teacher, a mimicked student is able to outperform it and achieves 51.6 times compression ratio and 90 times speed-up in inference, making this cumbersome model applicable on portable devices.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “h as face recognition and action recognition. The existing KD-based face recognition methods can not only easily solve deployment of deep neural networks, but also improve the classiﬁcation accuracy (Luo et al., 2016; Kong et al., 2019; Yan et al., 2019; Ge et al., 2018;Wang et al.,2018b,2019c; Duon”
Relational Knowledge Distillation (2019, methodology): “With growing interests in KD, task-specific KD methods have been proposed for object detection [5, 6, 37], face model compression [24], and image retrieval and Re-ID [7].”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “Luo et al. (2016) show that the knowledge of the teacher can also be obtained from the neurons in the top hidden layer.”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, result): “Different from previous works which represented the knowledge using the soften label probabilities, [54] represented the knowledge by using the neurons in the higher hidden layer, which preserved as much information as the label probabilities, but are more compact.”
Improved Knowledge Distillation for Training Fast Low Resolution Face Recognition Model (2019, methodology): “Among the three losses, the soft loss can transfer the knowledge from teacher to student by using soft label, the hard loss can make the student develop its own classification ability, and the hint learning can boost the performance and accelerate the convergence according to [6].”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, methodology): “Variants of knowledge distillation has also been applied to tasks in other domains [5, 23, 30, 33] A recent preprint [17] presented promising preliminary results on CIFAR-10 by learning a small ResNet from a large ResNet.”
MobileFaceNets: Efficient CNNs for Accurate Real-time Face Verification on Mobile Devices (2018, background): “In [17], a compact student network (denoted as MobileID) trained by distilling knowledge from the teacher network DeepID2+ [33] achieves 97.32% accuracy on LFW with 4.0MB model size.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, methodology): “A widely-used method for model compression is knowledge distillation, which matches the distribution of the output layer’s logits [23, 43, 9, 63, 33, 46, 10].”
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, methodology): “Knowledge distillation has been exploited for many computer vision tasks such as domain adaptation [15], object detection [1], learning from noisy labels [24] or facial analysis [26].”
Learning an Evolutionary Embedding via Massive Knowledge Distillation (2020, methodology): “[41] propose a neuron selection method to compress models by the teacher network.”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “Some other methods leverage knowledge distillation (Hinton et al., 2015; Romero et al., 2014; Luo et al., 2016; Chen et al., 2015a) to obtain a small and compact model with distilled knowledge of a larger one.”
A review of convolutional neural network architectures and their optimizations (2022, methodology): “used the output of high-level neurons of teacher network to represent the domain knowledge needing to be transferred in 2016 (Luo et al. 2016).”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “works which represented the knowledge using the soften label probabilities, [54] represented the knowledge by using the neurons in the higher hidden layer, which preserved as much information as the label probabilities, but are more compact.”
Fast Generalized Distillation for Semi-Supervised Domain Adaptation (2017, methodology): “…GDSDA-SVM, we use Mean Squared Error (MSE) as our loss function for the following two reasons: (1) several recently studies (Ba and Caruana 2014; Luo et al. 2016; Romero et al. 2015; Urban et al. 2016) show that MSE is also an efficient measurement for the target model to mimick the output of…”
Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification (2017, background): “In distillation, except for the available soft predictions PT(classification target), another one is the task-specific target that can be the hidden layer KT[16], and it satisfies RT = fc (KT) with fc being a fully-connected mapping.”
Discover the Effective Strategy for Face Recognition Model Compression by Improved Knowledge Distillation (2018, methodology): “Among the three losses, the soft loss can distill the knowledge from teacher to student by using soft label, the hard loss can make the student develop its own classification ability, and the hint learning can boost the performance and accelerate the convergence according to [3, 10].”
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation (2019, methodology): “Although training the very deep network on such enormous datasets to achieve satisfied performance has been a solvable issue, how to obtain the comparable performance with a tiny network by the knowledge distillation still confuses the research community, especially for the methods [22, 26, 34] with”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, methodology): “Note that it differs from many teacher-student frameworks [16]–[18], [20], [41], [42], where training is restricted on the private dataset or the same resolution or available target dataset.”