Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

By Hayato Futami, Hirofumi Inaguma, Sei Ueno, Masato Mimura, Shinsuke Sakai, Tatsuya Kawahara. 2020

59 citations recorded by the source corpus.

Attention-based sequence-to-sequence (seq2seq) models have achieved promising results in automatic speech recognition (ASR). However, as these models decode in a left-to-right way, they do not have access to context on the right. We leverage both left and right context by applying BERT as an external language model to seq2seq ASR through knowledge distillation. In our proposed method, BERT generates soft labels to guide the training of seq2seq ASR. Furthermore, we leverage context beyond the current utterance as input to BERT. Experimental evaluations show that our method significantly improves the ASR performance from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ). Knowledge distillation from BERT outperforms that from a transformer LM that only looks at left context. We also show the effectiveness of leveraging context beyond the current utterance. Our method outperforms other LM application approaches such as n-best rescoring and shallow fusion, while it does not require extra inference cost.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Tara N. Sainath
3. Dong Yu
4. Ron J. Weiss
5. Li Deng
6. Peidong Wang
7. Yu Zhang
8. Shinji Watanabe

In Efficiency & Compression

1. Jinyu Li
2. Ron J. Weiss
3. Tara N. Sainath
4. Dong Yu

In Speech & Audio

1. Jinyu Li
2. Tara N. Sainath
3. Ron J. Weiss
4. Peidong Wang
5. Dong Yu
6. Yu Zhang
7. Yonghui Wu
8. Shinji Watanabe

Citations in the corpus

• Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “As an alternative to LM fusion and TTS, knowledge distillation methods were proposed to transfer the knowledge in an LM to the ASR model [26, 27].”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [14, 65], an LM trained on large scale text is used as a teacher model to generate soft labels as the regularization of the AED model training.”