KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition

By Dong Yu, Kaisheng Yao, Hang Su, Gang Li, Frank Torsten Bernd Seide. 2013

455 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Yoshua Bengio
5. Tara N. Sainath
6. Khe Chai Sim
7. Shinji Watanabe
8. Florian Metze

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Khe Chai Sim
5. Shinji Watanabe
6. Tara N. Sainath
7. Yifan Gong
8. Florian Metze

Citations in the corpus

• Deep Learning: Methods and Applications (2014, methodology): “The most notable groups include major industrial speech labs worldwide: Microsoft [49, 89, 94, 324, 399, 430], IBM [195, 309, 311, 307, 317], Google [69, 150, 184, 223], iFlyTek, and Baidu.”
• Neural Voice Cloning with a Few Samples (2018, methodology): “The first group is speaker adaptation for the whole-model [Yu et al., 2013], a portion of the model [Miao and Metze, 2015, Cui et al., 2017], or merely to a speaker embedding [Abdel-Hamid and Jiang, 2013, Xue et al., 2014].”
• Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation (2018, methodology): “We draw inspiration from a number of prior works including Yu et al. (2013), which introduces Kullback-Leibler (KL) divergence between the model being trained and an out-of-domain model as a regularization scheme for speaker adaptation.”
• Continual Adaptation for Efficient Machine Communication (2019, unknown): “Specifically, we consider a KL regularization term that explicitly minimizes the divergence between the captioning model’s output probabilities before and after finetuning for unseen images (Yu et al., 2013; Galashov et al., 2018).”
• Fast Adaptation of Deep Neural Network Based on Discriminant Codes for Speech Recognition (2014, unknown): “Moreover, the similar transformation functions may be directly applied in the feature space to trans­form speech features towards any given speaker-independent HMM models [5], [6].”
• Recent progresses in deep learning based acoustic models (2017, methodology): “[99] adds Kullback-Leibler divergence (KLD) regularization to the training criterion to prevent the adapted model from straying too far away from the SI model.”
• Speaker adaptation of context dependent deep neural networks based on MAP-adaptation and GMM-derived feature processing (2014, methodology): “• Re-training the entire network or only input or output layers of the DNN using special regularization techniques for improving generalization, such as L2-prior regularization [14], Kullback-Leibler divergence regularization [15], Conservative Training [16].”
• Maximum a posteriori adaptation of network parameters in deep models (2015, methodology): “Recent approaches, such as regularization based [7, 8], subspace based [9, 10], transformation based [6, 11, 12, 13], i-Vector based [14], native neural network based [15, 16, 17], factorization based [18] and fast adaptation schemes based on discriminant speaker codes [19, 16, 20, 21], have been pr”
• Conditional Teacher-student Learning (2019, methodology): “As a comparison, the standard adaptation with hard labels and KLD adaptation [25] with regularization weights λ of 0.”
• Domain Adaptation Using Factorized Hidden Layer for Robust Automatic Speech Recognition (2018, methodology): “Additional loss term, such as the KL-divergence between the outputs of the original and fine-tuned models [18], can be used to ensure that the fine-tuned model does not deviate significantly from the original model.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “The first one is using regularization techniques such as Kullback-Leibler (KL) divergence regularization [169], maximum a posteriori adaptation [170], or elastic weight consolidation [171], to ensure the adapted models do not overfit the limited adaptation data [160, 162–164].”
• Multi-domain Neural Network Language Generation for Spoken Dialogue Systems (2016, unknown): “However, careful regularisation is often necessary (Yu et al., 2013).”
• Cluster adaptive training for deep neural network (2015, methodology): “maximum a posteriori (MAP) adaptation, the KL-divergence regularization adaptation uses the KL distance to keep the adapted DNN close to a well trained unadapted DNN [8].”
• Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “e not using cases), and (2) when query and cases (Dq) are available (Yu et al., 2013).”