Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition

By Ye Bai, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Zhengqi Wen. 2019

40 citations recorded by the source corpus.

Integrating an external language model into a sequence-to-sequence speech recognition system is non-trivial. Previous works utilize linear interpolation or a fusion network to integrate external language models. However, these approaches introduce external components, and increase decoding computation. In this paper, we instead propose a knowledge distillation based training approach to integrating external language models into a sequence-to-sequence model. A recurrent neural network language model, which is trained on large scale external text, generates soft labels to guide the sequence-to-sequence model training. Thus, the language model plays the role of the teacher. This approach does not add any external component to the sequence-to-sequence model during testing. And this approach is flexible to be combined with shallow fusion technique together for decoding. The experiments are conducted on public Chinese datasets AISHELL-1 and CLMAD. Our approach achieves a character error rate of 9.3%, which is relatively reduced by 18.42% compared with the vanilla sequence-to-sequence model.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “As an alternative to LM fusion and TTS, knowledge distillation methods were proposed to transfer the knowledge in an LM to the ASR model [26, 27].”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, background): “Prior work focuses on building LM with monolingual text data, and then integrate LM or transfer knowledge from it into the decoder [13, 15, 16].”
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR (2020, background): “More recently, knowledge distillation [10] -based LM integration has been proposed [11].”
Rnn-Transducer with Stateless Prediction Network (2020, methodology): “Many recent works have investigated effective ways of using text-only data to improve E2E models, including fusion with an external LM [4, 5, 6, 7, 8], knowledge distillation from an LM [9], augmenting paired data with TTS data [10, 11], etc.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, methodology): “Inspired by [2], our introduced TRL method only calculates the KL divergence between “soft targets” and output distribution of task model during training which can well overcome above-mentioned limitations.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [14, 65], an LM trained on large scale text is used as a teacher model to generate soft labels as the regularization of the AED model training.”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “Another approach uses an RNN-LM, trained on text-only data, as the teacher model to regularize the label distribution of the E2E model while training on paired data [13].”
Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, background): “Several of these studies focused on building a language model (LM) with text data and then integrating the LM with or transferring its knowledge to the ASR decoder [6, 7].”