Search to Distill: Pearls Are Everywhere but Not the Eyes

By Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, Xiaogang Wang. 2019

73 citations recorded by the source corpus.

Standard Knowledge Distillation (KD) approaches distill the knowledge of a cumbersome teacher model into the parameters of a student model with a pre-defined architecture. However, the knowledge of a neural network, which is represented by the network's output distribution conditioned on its input, depends not only on its parameters but also on its architecture. Hence, a more generalized approach for KD is to distill the teacher's knowledge into both the parameters and architecture of the student. To achieve this, we present a new \textit{Architecture-aware Knowledge Distillation (AKD)} approach that finds student models (pearls for the teacher) that are best for distilling the given teacher model. In particular, we leverage Neural Architecture Search (NAS), equipped with our KD-guided reward, to search for the best student architectures for a given teacher. Experimental results show our proposed AKD consistently outperforms the conventional NAS plus KD approach, and achieves state-of-the-art results on the ImageNet classification task under various latency settings. Furthermore, the best AKD student architecture for the ImageNet classification task also transfers well to other tasks such as million level face recognition and ensemble learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Qiang Yang
3. Han Yu
4. Aditya Krishna Menon
5. Jianping Gou
6. Sanjiv Kumar
7. S. Maybank
8. Baosheng Yu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianlong Fu
3. Qiang Yang
4. Houwen Peng
5. Jianping Gou
6. Luc Van Gool
7. S. Maybank
8. Baosheng Yu

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
• Towards Personalized Federated Learning (2021, background): “As it may be difficult for the student model to learn well if there is a huge capacity gap between the large teacher model and the small student model [73, 74], it is imperative to determine an optimal design for both the server and client models.”
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, background): “6 [26] Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang.”
• Self-Distillation: Towards Efficient and Compact Neural Networks (2021, background): “Reinforcement learning based technique is also presented to search for the best teacher model for knowledge distillation [46].”
• Task-Oriented Feature Distillation (2020, background): “have applied neural network searching to find the student models for a given teacher model [31, 38].”
• A statistical perspective on distillation (2021, background): “, 2018), and found several broader uses (Papernot et al., 2016; Yim et al., 2017; Liu et al., 2019).”