Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition

By Gakuto Kurata, George Saon. 2020

55 citations recorded by the source corpus.

End-to-end training of recurrent neural network transducers (RNN-Ts) does not require frame-level alignments between audio and output symbols. Because of that, the posterior lattices defined by the predictive distributions from different RNN-Ts trained on the same data can differ a lot, which poses a new set of challenges in knowledge distillation between such models. These discrepancies are especially prominent in the posterior lattices between an offline model and a streaming model, which can be expected from the fact that the streaming RNN-T emits symbols later than the offline RNN-T. We propose a method to train an RNN-T so that the posterior peaks at each node in the posterior lattice are aligned with the ones from a pretrained model for the same utterance. By utilizing this method, we can train an offline RNN-T that can serve as a good teacher to train a student streaming RNN-T. Experimental results on the standard Switchboard conversational telephone speech corpus demonstrate accuracy improvements for a streaming unidirectional RNN-T by knowledge distillation from an offline bidirectional counterpart.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yu Zhang
2. Wei Han
3. Jinyu Li
4. Chung-Cheng Chiu
5. Bo Li
6. Thibault Doutre
7. Ruoming Pang
8. Arun Narayanan

Citations in the corpus

• Recent Advances in End-to-End Automatic Speech Recognition (2021, unknown): “Therefore, delicate work has been done in order to make the non-streaming E2E models generate friendly output spikes that can be learned by the streaming E2E models [112, 114].”
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Some work attempts on improving streaming ASR using model distillation [14, 15, 16, 17].”
• Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, methodology): “[12] also proposed to use a full-context model as the teacher for RNN-T models, but their approach uses posteriors as targets and therefore requires additional distillation pretraining to address the alignment mismatch between the student and the teacher.”