Improved training for online end-to-end speech recognition systems
By Suyoun Kim, M. Seltzer, Jinyu Li, Rui Zhao. 2017
47 citations recorded by the source corpus.
Achieving high accuracy with end-to-end speech recognizers requires careful parameter initialization prior to training. Otherwise, the networks may fail to find a good local optimum. This is particularly true for online networks, such as unidirectional LSTMs. Currently, the best strategy to train such systems is to bootstrap the training from a tied-triphone system. However, this is time consuming, and more importantly, is impossible for languages without a high-quality pronunciation lexicon. In this work, we propose an initialization strategy that uses teacher-student learning to transfer knowledge from a large, well-trained, offline end-to-end speech recognition model to an online end-to-end model, eliminating the need for a lexicon or any other linguistic resources. We also explore curriculum learning and label smoothing and show how they can be combined with the proposed teacher-student learning for further improvements. We evaluate our methods on a Microsoft Cortana personal assistant task and show that the proposed method results in a 19 % relative improvement in word error rate compared to a randomly-initialized baseline system.
Subfields: Artificial Intelligence, Optimization & Training, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Optimization & Training
In Speech & Audio
In Signal Processing
Citations in the corpus
- Recent Advances in End-to-End Automatic Speech Recognition (2021, unknown): “Another usage is to learn a streaming E2E model from a non-streaming E2E model [130, 134, 136, 137, 194].”
- HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis (2022, unknown): “For better generalization, we attempted to combine label smoothing with CTC loss (Kim et al., 2018) and conduct experiments on data augmentation for the effective speech disentanglement of linguistic information following (Choi et al., 2021).”
- Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, methodology): “Because of its more simplified training pipeline and faster decoding speed [13], a number of studies have considered deploying E2E ASR using CTC in actual products and services [14, 15].”