Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model

By Bo Li, Tara N. Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Z. Chen, Yang Wu, Kanishka Rao. 2017

140 citations recorded by the source corpus.

Sequence-to-sequence models provide a simple and elegant solution for building speech recognition systems by folding separate components of a typical system, namely acoustic (AM), pronunciation (PM) and language (LM) models into a single neural network. In this work, we look at one such sequence-to-sequence model, namely listen, attend and spell (LAS) [1], and explore the possibility of training a single model to serve different English dialects, which simplifies the process of training multi-dialect systems without the need for separate AM, PM and LMs for each dialect. We show that simply pooling the data from all dialects into one LAS model falls behind the performance of a model fine-tuned on each dialect. We then look at incorporating dialect-specific information into the model, both by modifying the training targets by inserting the dialect symbol at the end of the original grapheme sequence and also feeding a 1-hot representation of the dialect information into all layers of the model. Experimental results on seven English dialects show that our proposed system is effective in modeling dialect variations within a single LAS model, outperforming a LAS model trained individually on each of the seven dialects by 3.1~16.5% relative.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Dong Yu
4. Tara N. Sainath
5. Shinji Watanabe
6. Yonghui Wu
7. Quoc V. Le
8. Yoshua Bengio

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Shinji Watanabe
5. Yonghui Wu
6. Wei-Ning Hsu
7. Chung-Cheng Chiu
8. Dong Yu

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Yu Zhang
5. Shinji Watanabe
6. Yonghui Wu
7. Chung-Cheng Chiu
8. Wei-Ning Hsu

Citations in the corpus

• Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “Apart from speaking style, other speech variabilities like speaking environment, speech scenarios and regional dialect [26, 30, 31] also have an impact on the performance of speech recognition system.”
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
• Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, background): “In addition, a simple 1-hot language ID vector has been found to be effective improving multilingual systems [24, 25].”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “If the multilingual model can condition on the language identity (LID), significant improvement can be obtained over the universal multilingual model without LID because the one-hot LID guides the ASR model to generate the transcription of the target language by reducing the confusion from other lan”
• Speechmoe2: Mixture-of-Experts Model with Improved Routing (2021, background): “[12] has shown that incorporating global dialectspecific information(e.”
• Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “promising capabilities of such E2E models in modeling dialects of a particular language [16], languages from the same family [17] and languages from different families [18–21].”
• Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, methodology): “Bolstering the decoder with a language model (LM) trained on supplemental text data is one such method that improves end-to-end ASR performance [4, 5]; however, more significant gains can be obtained by training on additional synthetically perturbed speech [6, 7, 8], or by multilingual training, whi”
• An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “ASRU, 2017, pp. 193–199.”
• Multilingual Speech Recognition with Corpus Relatedness Sampling (2019, background): “For instance, the language level (corpus level) embedding can be generated along with the model in machine translation [17] and speech recognition [18].”
• Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, background): “[14] adds a special accent-specific symbol at the end of target transcriptions to train a multidialect sequence-to-sequence model.”
• Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model (2019, methodology): “The language itself can be represented in several different ways (as a one-hot vector, as an embedding vector, or as a combination of clusters learned through cluster adaptive training (CAT) [23]), but prior work [18, 19] has shown that the simple approach of a one-hot vector performs as well as and”