Student-teacher network learning with enhanced features
By Shinji Watanabe, Takaaki Hori, Jonathan Le Roux, John R. Hershey. 2017
89 citations recorded by the source corpus.
Subfields: Artificial Intelligence, Computational Mechanics, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computational Mechanics
In Signal Processing
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “…et al. 2018, 2019c, 2020), audio classification (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic event detection (Price et al. 2016; Shi et al. 2019a,b), speech synthesis (Oord et al. 2018) and so on.”
- Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classification [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
- An Investigation of a Knowledge Distillation Method for CTC Acoustic Models (2018, methodology): “The effectiveness of KD has been confirmed in speech recognition tasks [20, 21, 22, 23].”
- Improved training for online end-to-end speech recognition systems (2017, background): “This approach has been shown to be successful in several tasks including acoustic modeling [26], speech enhancement [27], and domain adaptation [28].”
- Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training (2020, background): “A typical application is to transfer knowledge from a large, powerful teacher network to a compact yet accurate student network, so as to boost the training process and the resulting performance (Watanabe et al., 2017; Bucilu and Niculescu-Mizil, 2006; Wang et al., 2018).”
- Large-Scale Domain Adaptation via Teacher-Student Learning (2017, methodology): “In [14] it was shown that for the multi-channel CHiME-4 task [15], soft labels could be derived using enhanced features generated by a beamformer then processed through a network trained with conventional multi-style training [16].”
- Recent progresses in deep learning based acoustic models (2017, methodology): “In [164] it was shown that for the multi-channel CHiME-4 task, soft labels could be derived using enhanced features generated by a beamformer then processed through a network trained with conventional multi-style training.”
- Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “Different from previous work applying T-S learning for speech enhancement and separation, which train student models in the same model size while operating at different input features [8, 9], this paper aims to distill the teacher model’s knowledge to create a smaller and faster student model.”
- Conditional Teacher-student Learning (2019, background): “But in speech area, T is normally fixed at 1 [9, 13, 14].”
- Knowledge Distillation for Sequence Model (2018, background): “Although large amount of un-transcribed data can be helpful [13], discarding transcription can hamper the performance and requires additional steps to utilize the transcription during [14] or after knowledge distillation [9, 11].”
- Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, methodology): “Current successful ASR systems employ deep neural network (DNN) models as acoustic model combined with a hidden Markov model (HMM) [1, 2], or use them within an endto-end configuration [3, 4].”