Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition

By Gakuto Kurata, Kartik Audhkhasi. 2018

44 citations recorded by the source corpus.

End-to-end automatic speech recognition (ASR) promises to simplify model training and deployment. Most end-to-end ASR systems utilize a bi-directional Long Short-Term Memory (BiLSTM) acoustic model due to its ability to capture acoustic context from the entire utterance. However, BiLSTM models have high latency and cannot be used in streaming applications. Leveraging knowledge distillation to train a low-latency end-to-end uni-directional LSTM (UniLSTM) model from a BiLSTM model can be an option. However, it makes the strict assumption of shared frame-wise time alignments between the two models. We propose an improved knowledge distillation algorithm that relaxes this assumption and improves the accuracy of the UniLSTM model. We confirmed the advantage of the proposed method on a standard English conversational telephone speech recognition task.

Subfields: Artificial Intelligence, Optimization & Training, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Han
2. Ruoming Pang
3. Chung-Cheng Chiu
4. Liangliang Cao
5. Jinyu Li
6. Thibault Doutre
7. Weiran Wang
8. Yu Zhang

In Optimization & Training

1. Wei Han
2. Chung-Cheng Chiu
3. Yang Chen
4. Weiran Wang
5. Ruoming Pang
6. Liangliang Cao
7. Chao Wang

In Speech & Audio

1. Chung-Cheng Chiu
2. Wei Han
3. Ruoming Pang
4. Liangliang Cao
5. Weiran Wang
6. Yang Chen
7. Olivier Siohan
8. Chao Wang

In Signal Processing

1. Ruoming Pang
2. Wei Han
3. Chung-Cheng Chiu
4. Liangliang Cao
5. Thibault Doutre
6. Weiran Wang
7. Yang Chen
8. Olivier Siohan

Citations in the corpus

• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Some work attempts on improving streaming ASR using model distillation [14, 15, 16, 17].”
• Semi-supervised ASR by End-to-end Self-training (2020, result): “The observation that hard labels outperform soft targets is in line with that of [33] for teacher-student learning with CTC.”