Knowledge Distillation via Route Constrained Optimization

By Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, Xiaolin Hu. 2019

203 citations recorded by the source corpus.

Distillation-based learning boosts the performance of the miniaturized neural network based on the hypothesis that the representation of a teacher model can be used as structured and relatively weak supervision, and thus would be easily learned by a miniaturized model. However, we find that the representation of a converged heavy model is still a strong constraint for training a small student model, which leads to a higher lower bound of congruence loss. In this work, we consider the knowledge distillation from the perspective of curriculum learning by teacher's routing. Instead of supervising the student model with a converged teacher model, we supervised it with some anchor points selected from the route in parameter space that the teacher model passed by, as we called route constrained optimization (RCO). We experimentally demonstrate this simple operation greatly reduces the lower bound of congruence loss for knowledge distillation, hint and mimicking learning. On close-set classification tasks like CIFAR and ImageNet, RCO improves knowledge distillation by 2.14% and 1.5% respectively. For the sake of evaluating the generalization, we also test RCO on the open-set face recognition task MegaFace. RCO achieves 84.3% accuracy on one-to-million task with only 0.8 M parameters, which push the SOTA by a large margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Localization Distillation for Object Detection (2022, background): “It has been widely validated to be useful for boosting the performance of small-sized student networks by transferring the generalized knowledge captured by large-sized teacher networks [24], [29], [31], [58], [63], [78].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Among the works focusing on feature-based distillation, KSANC [99] adds an adversarial loss at the last layer of both teacher and student networks, while MEAL [198] adds multi-stage discriminators in the position of every extracted feature representation.”
Curriculum Temperature for Knowledge Distillation (2022, methodology): “RCO (Jin et al. 2019) proposes to utilize the sequence of teachers’ intermediate states as a curriculum to supervise the student at different learning stages.”
Search to Distill: Pearls Are Everywhere but Not the Eyes (2019, methodology): “So we verify it by transferring the Pareto optimal architecture at ∼33 ms (denoted as AKDNet-M) to the three recent state-of-the-art KD methods, TA-KD [21], RCO-KD [13] and CC-KD [22].”
Spiking Neural Networks and Their Applications: A Review (2022, background): “In order to ﬁnd more compact networks that can achieve similar performance with much less complexity and a smaller number of parameters compared to the original network, many techniques have been developed such as quantization [1], pruning [2], and knowledge distillation [3].”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “However, the teacher in this framework changes smoothly, whereas that in [15] changes discretely.”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “For face recognition model compression, there have been several attempts [16, 40, 51, 26, 19, 52, 18, 35, 9, 10] in literatures to distil large CNNs, so as to make their deployments easier.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Jin et al. present the RCO distillation to employ various teacher models with different accuracy rates for a single student model [44].”