Teaching Semi-Supervised Classifier via Generalized Distillation

By Chen Gong, Xiaojun Chang, Meng Fang, Jian Yang. 2018

28 citations recorded by the source corpus.

Semi-Supervised Learning (SSL) is able to build reliable classifier with very scarce labeled examples by properly utilizing the abundant unlabeled examples. However, existing SSL algorithms often yield unsatisfactory performance due to the lack of supervision information. To address this issue, this paper formulates SSL as a Generalized Distillation (GD) problem, which treats existing SSL algorithm as a learner and introduces a teacher to guide the learner?s training process. Specifically, the intelligent teacher holds the privileged knowledge that ?explains? the training data but remains unknown to the learner, and the teacher should convey its rich knowledge to the imperfect learner through a specific teaching function. After that, the learner gains knowledge by ?imitating? the output of the teaching function under an optimization framework. Therefore, the learner in our algorithm learns from both the teacher and the training data, so its output can be substantially distilled and enhanced. By deriving the Rademacher complexity and error bounds of the proposed algorithm, the usefulness of the introduced teacher is theoretically demonstrated. The superiority of our algorithm to the related state-of-the-art methods has also been empirically demonstrated by the experiments on different datasets with various sources of privileged knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. J. Lai
3. Jianping Gou
4. Weishi Zheng
5. S. Maybank
6. Baosheng Yu
7. Lin Wang
8. Kuk-Jin Yoon

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. J. Lai
3. Weishi Zheng
4. Jianping Gou
5. S. Maybank
6. Ancong Wu
7. Xiaogang Wang
8. Baosheng Yu

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “It can be practicable to popularize the knowledge transfer to the classic and traditional machine learning methods (Zhou et al. 2019b; Gong et al. 2018; You et al. 2018; Gong et al. 2017).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “To tackle these problems, some recent works [64], [182], [191], [219], [231], [234] propose data distillation by unifying data sources from multiple teachers as illustrated in Fig.”
• Distilled Person Re-Identification: Towards a More Scalable System (2019, background): “Semi-supervised teacher-student frameworks [48, 17] are for closed-set classification and cannot solve our problem.”