Sequence Student-Teacher Training of Deep Neural Networks

By Jeremy Heng Meng Wong, Mark John Francis Gales. 2016

96 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “To overcome the degradation of the performance of non-native speech recognition, an advanced multi-accent student model is trained by distilling knowledge from the multiple accent-speciﬁc RNN-CTC models (Ghorbani et al. 2018).”
Neural Machine Translation: A Review (2019, background): “Wu et al. (2016b) proposed the following term:”
Efficient Video Classification Using Fewer Frames (2019, background): “This idea of Knowledge Distillation has also been tried in the context of pruning networks for multiple object detection [4], speech recognition [38] and reading comprehension [14].”
Have Best of Both Worlds: Two-Pass Hybrid and E2E Cascading Framework for Speech Recognition (2021, methodology): “• STEP1: Given the audio and transcription pairs, the first pass hybrid AM is trained with a 3-stage model training recipe [26]: cross-entropy (CE), maximum mutual information (MMI) [27], as well as sequence-level teacher-student (T/S) learning [28].”
Efficient Knowledge Distillation from an Ensemble of Teachers (2017, methodology): “This technique has been shown to be very successful in various settings - fully supervised [9], semi-supervised [6], multilingual [11], sequence training [12] - to train student networks perform better than training similar models from scratch using hard targets.”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, background): “The full teacher sequence distribution is often intractable, but it can be approximated with a lattice [23] for hybrid or the top k hypotheses for CTC [24] and seq2seq [25, 26].”
Knowledge Distillation for Sequence Model (2018, methodology): “[17] constrains teacher hypotheses by n-best list from beam search, which can hinder the improvement.”
On Knowledge distillation from complex networks for response prediction (2019, unknown): “This idea of Knowledge Distil-lation has also been tried in the context of pruning networks for multiple object detection (Chen et al., 2017), speech recognition (Wong and Gales, 2016).”
Recent progresses in deep learning based acoustic models (2017, background): “…T/S learning is essentially a frame-by-frame CE training criterion with soft labels, it may be desirable to develop a sequence-based T/S criterion [174], [175] if the teacher model was trained with sequence criterion although in practice the standard T/S learning works very well even in such a…”
Teacher-Student Training for Text-Independent Speaker Recognition (2018, background): “Such application scenario has not been investigated thoroughly in the literature on SR tasks, but was found promising in related areas like speech recognition [16, 18, 19].”
Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, background): “Knowledge distillation is an active area of research [5, 6, 7, 8, 9, 10].”