An Investigation of a Knowledge Distillation Method for CTC Acoustic Models

By R. Takashima, Sheng Li, Hisashi Kawai. 2018

67 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Dacheng Tao
4. Chung-Cheng Chiu
5. Kartik Audhkhasi
6. Ruoming Pang
7. Gakuto Kurata
8. Wei Han

In Efficiency & Compression

1. Dacheng Tao
2. Jinyu Li
3. Gakuto Kurata
4. Kartik Audhkhasi
5. Chung-Cheng Chiu
6. Jianping Gou
7. Wei Han
8. Oncel Tuzel

In Speech & Audio

1. Jinyu Li
2. Chung-Cheng Chiu
3. Wei Han
4. Gakuto Kurata
5. Kartik Audhkhasi
6. Ruoming Pang
7. M. Seltzer
8. Rui Zhao

In Signal Processing

1. Jinyu Li
2. Kartik Audhkhasi
3. Gakuto Kurata
4. Ruoming Pang
5. Dong Yu
6. Chung-Cheng Chiu
7. Wei Han
8. M. Seltzer

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “…Ng et al. 2018; Albanie et al. 2018; Lu et al. 2017; Shi et al. 2019a; Roheda et al. 2018; Shi et al. 2019b; Gao et al. 2019; Ghorbani et al. 2018; Takashima et al. 2018; Watan-abe et al. 2017; Shi et al. 2019c; Asami et al. 2017; Huang et al. 2018; Shen et al. 2018; Perez et al. 2020; Shen et…”
• Semi-supervised ASR by End-to-end Self-training (2020, background): “Furthermore, for larger W one may consider the top a few hypotheses, and use all of them for computing the loss on unsupervised data [38, 25].”
• Improved training for online end-to-end speech recognition systems (2017, methodology): “e aim to improve the accuracy of a unidirectional online model by transferring knowledge from a bidirectional offline model that does not have any latency or complexity constraints. The recent work in [29] proposed a similar teacher-student method and they use the criteria as an additional objective.”
• Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, methodology): “5[18] also reported that WER deteriorated as a result of simply applying the naı̈ve KD from BiLSTM CTC to UniLSTM CTC.”
• Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been implemented in many machine learning tasks, for example image classification [25], object detection [10, 61], video labeling [69], natural language processing [57, 38, 54, 34, 58], and speech recognition [9, 56, 35].”
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Some work attempts on improving streaming ASR using model distillation [14, 15, 16, 17].”