Learning With Single-Teacher Multi-Student

By Shan You, Chang Xu, Chao Xu, Dacheng Tao. 2018

29 citations recorded by the source corpus.

In this paper we study a new learning problem defined as "Single-Teacher Multi-Student" (STMS) problem, which investigates how to learn a series of student (simple and specific) models from a single teacher (complex and universal) model. Taking the multiclass and binary classification for example, we focus on learning multiple binary classifiers from a single multiclass classifier, where each of binary classifier is responsible for a certain class. This actually derives from some realistic problems, such as identifying the suspect based on a comprehensive face recognition system. By treating the already-trained multiclass classifier as the teacher, and multiple binary classifiers as the students, we propose a gated support vector machine (gSVM) as a solution. A series of gSVMs are learned with the help of single teacher multiclass classifier. The teacher's help is two-fold; first, the teacher's score provides the gated values for students' decision; second, the teacher can guide the students to accommodate training examples with different difficulty degrees. Extensive experiments on real datasets validate its effectiveness.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Chen Qian
3. Fei Wang
4. Jianlong Wu
5. Xiaojie Li
6. Shan You
7. Jianping Gou
8. Changshui Zhang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Fei Wang
3. Chen Qian
4. Jianlong Wu
5. Changshui Zhang
6. Yue Liao
7. Shan You
8. Jianping Gou

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “It can be practicable to popularize the knowledge transfer to the classic and traditional machine learning methods (Zhou et al. 2019b; Gong et al. 2018; You et al. 2018; Gong et al. 2017).”
• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, methodology): “Inspired by the teacher-student paradigm [17, 40, 42] in human education, knowledge distillation (KD) [9] serves as a complementary method to further boost the performance of a small student network by distilling the knowledge from a large teacher network.”
• Local Correlation Consistency for Knowledge Distillation (2020, methodology): “Typical methods include network pruning [6, 17, 4], compact architecture design [9, 40], network quantization [5, 31, 11], knowledge distillation [8, 41, 35, 36, 1, 26], and so on.”
• Adaptive multi-teacher multi-level knowledge distillation (2020, background): “AvgMKD [29] : This approach distills knowledge from multiple teachers by: 1) treating each teacher equally when calculating integrated soft-targets; and 2) transferring relative dissimilarity (RD) relationships among the intermediate representations of teacher networks.”