Efficient Knowledge Distillation from an Ensemble of Teachers

By Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, Bhuvana Ramabhadran. 2017

280 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Fukuda et al. (2017) randomly selected one teacher from the pool of teacher networks at each iteration.”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classiﬁcation [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, methodology): “This technique has achieved success in conventional DNN-HMM hybrid systems [21, 22].”
Distilled Person Re-Identification: Towards a More Scalable System (2019, methodology): “[16] also exploited multiple teachers, but it is based on soft labels for closed-set classification, which is not effective for the open-set Re-ID problem.”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “Many papers that related to knowledge distillation are often used to train a student network by averaged the soft label from multiple cumbersome models (Fukuda et al., 2017).”
Adaptive Knowledge Distillation Based on Entropy (2020, methodology): “…teacher models are utilized, smoothed labels, , for the KD loss are generally obtained as follows: 𝑡𝑡 = ∑ 𝑤𝑤 𝑡𝑡 𝑡𝑡 where 𝑁𝑁 , 𝑤𝑤 , and 𝑡𝑡 𝑇𝑇 𝑖𝑖 are the number of teacher models, the weight for the -th teacher model, and the soft labels of the -th teacher model, respectively [13].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Different from the methods as mentioned above, [56], [114], [185], [234], [263] argue that taking the average of individual prediction may ignore the diversity and importance variety of the member teachers of an ensemble.”
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition (2019, background): “Such a simple yet effective idea inspires many researchers and has been further studied by [8, 10, 38].”
An Investigation of a Knowledge Distillation Method for CTC Acoustic Models (2018, methodology): “[23] showed that KD can improve the word error rate (WER) of a student convolutional neural network (CNN) using a VGG network [24] and a long short-term memory (LSTM) network [25] as teacher models.”
Adaptive multi-teacher multi-level knowledge distillation (2020, background): “The most relevant studies to this paper are as follows: [6] manually tunes the weights of each teacher, which is costly and not automatic.”
Reinforced Multi-Teacher Selection for Knowledge Distillation (2020, methodology): “Rand-Single-Ensemble Teacher uses the strategy of Fukuda et al. (2017) to randomly select one teacher from the teacher model candidates at mini-batch level to provide soft-targets for training student model.”
Distilling Object Detectors with Task Adaptive Regularization (2020, methodology): “Similar works are presented in [31,19,6] but are designed mainly for classifiers.”
Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, background): “As a general setting, having a trained complex model or ensemble of neural networks (teacher models), it is possible to achieve an improved single (smaller) student by constraining it to mimic the soft outputs produced by the teacher(s) [15, 16].”
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, methodology): “[ 24] used perturbed logits to simulate learning from multiple teachers while [6] argued to randomly select a teacher to strengthen the complementariness in the ensemble.”