Domain adaptation of DNN acoustic models using knowledge distillation

By Taichi Asami, Ryo Masumura, Yoshikazu Yamaguchi, Hirokazu Masataki, Yushi Aono. 2017

90 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Gabriel Synnaeve
3. Dacheng Tao
4. Ronan Collobert
5. Dong Yu
6. Li Deng
7. Qiantong Xu
8. Yifan Gong

In Representation & Self-Supervised Learning

1. Jinyu Li
2. Yifan Gong
3. Zhong Meng
4. Dacheng Tao
5. S. Maybank
6. Wonyong Sung
7. Sheng Li
8. Tatiana Likhomanenko

In Efficiency & Compression

1. Jinyu Li
2. Sungho Shin
3. Yifan Gong
4. Dacheng Tao
5. Yong Zhao
6. Yoonho Boo
7. Zhong Meng
8. Wonyong Sung

In Speech & Audio

1. Jinyu Li
2. Gabriel Synnaeve
3. Ronan Collobert
4. Qiantong Xu
5. Tatiana Likhomanenko
6. Jacob Kahn
7. Vineel Pratap
8. Alexei Baevski

In Signal Processing

1. Jinyu Li
2. Yifan Gong
3. Gabriel Synnaeve
4. Zhong Meng
5. Ronan Collobert
6. Dong Yu
7. Qiantong Xu
8. Yong Zhao

Citations in the corpus

• Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, methodology): “Distillation was used to improved transfer in [3], where the soft-target part of the distillation loss may help with regularization.”
• Knowledge Distillation: A Survey (2020, methodology): “ased CTC student model via frame-level KD and sequence-level KD (Takashima et al., 2018). Moreover, knowledge distillation can be used to solvesomespecialissuesin speechrecognition(Bai et al., 2019b; Asami et al., 2017; Ghorbani et al., 2018). To overcome overfitting issue of DNN acoustic models in t”
• Conditional Teacher-student Learning (2019, background): “Apparently, the KLD adaptation is a special case of the interpolated T/S learning [12], in which the SI model acts as a teacher, the SD model acts as a student, and both take the adaptation data as input.”
• Knowledge Distillation-Based Representation Learning for Short-Utterance Spoken Language Identification (2020, unknown): “of dialect speech recognition [18].”
• Large-Scale Domain Adaptation via Teacher-Student Learning (2017, background): “formulation in [18] which takes the same data for teacher and student networks.”
• Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “Leveraging the knowledge contained in previously trained networks has attracted a lot of attention in a variety of applications for model compression (Tang et al., 2016; Song et al., 2018; Asami et al., 2017; Wang et al., 2019).”
• Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (2021, unknown): “2017; Tian, Krishnan, and Isola 2019), crossdomain transfer learning (Orbes-Arteainst et al. 2019; Li et al. 2017; Asami et al. 2017), and continual learning (Li and Hoiem 2017; Hou et al.”
• Interactive Learning of Teacher-student Model for Short Utterance Spoken Language Identification (2019, methodology): “Knowledge distillation method has been already successfully applied on many tasks, such as speech recognition, image classification [15, 16].”
• Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, methodology): “Adapting the baseline model using outputs of the model itself as softoutputs [20] results in a +12.”
• Knowledge Distillation for Optimization of Quantized Deep Neural Networks (2019, background): “Leveraging the knowledge contained in previously trained networks has attracted attention in many applications for model compression [8, 9, 10, 11] and learning algorithms [12, 13, 14, 15].”