Conditional Teacher-student Learning

By Zhong Meng, Jinyu Li, Yong Zhao, Yifan Gong. 2019

116 citations recorded by the source corpus.

The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student model towards a suboptimal performance. To overcome this problem, we propose a conditional T/S learning scheme, in which a "smart" student model selectively chooses to learn from either the teacher model or the ground truth labels conditioned on whether the teacher can correctly predict the ground truth. Unlike a naive linear combination of the two knowledge sources, the conditional learning is exclusively engaged with the teacher model when the teacher model’s prediction is correct, and otherwise backs off to the ground truth. Thus, the student model is able to learn effectively from the teacher and even potentially surpass the teacher. We examine the proposed learning scheme on two tasks: domain adaptation on CHiME-3 dataset and speaker adaptation on Microsoft short message dictation dataset. The proposed method achieves 9.8% and 12.8% relative word error rate reductions, respectively, over T/S learning for environment adaptation and speaker-independent model for speaker adaptation.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Chen Chen
6. Silong Peng
7. Shaoyu Zhang
8. Xiyuan Hu

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “Recently, response-based knowledge has been further explored to address the information of ground-truth label as the conditional targets (Meng et al. 2019).”
• Balanced Knowledge Distillation for Long-tailed Learning (2021, background): “Apart from the final prediction [35], other types of knowledge, like the intermediate representations [41, 24, 15] or the relationships between different layers [52] or data samples [54, 37, 32], can also be used to guide the learning of the student model.”