A study on multilingual acoustic modeling for large vocabulary ASR

By Hui-Ching Lin, Li Deng, Dong Yu, Yifan Gong, A. Acero, Chin-Hui Lee. 2009

130 citations recorded by the source corpus.

We study key issues related to multilingual acoustic modeling for automatic speech recognition (ASR) through a series of large-scale ASR experiments. Our study explores shared structures embedded in a large collection of speech data spanning over a number of spoken languages in order to establish a common set of universal phone models that can be used for large vocabulary ASR of all the languages seen or unseen during training. Language-universal and language-adaptive models are compared with language-specific models, and the comparison results show that in many cases it is possible to build general-purpose language-universal and language-adaptive acoustic models that outperform language-specific ones if the set of shared units, the structure of shared states, and the shared acoustic-phonetic properties among different languages can be properly utilized. Specifically, our results demonstrate that when the context coverage is poor in language-specific training, we can use one tenth of the adaptation data to achieve equivalent performance in cross-lingual speech recognition.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
Multilingual Speech Recognition with a Single End-to-End Model (2017, background): “…continuous speech recognizers are severly limited [1] These challenges have meant that there has been a perennial interest in multilingual and cross-lingual models which allow for knowledge transfer across languages, and thus relieve burdensome data requirements [2, 3, 4, 5, 6, 7, 8, 9, 10, 11].”
Deep Learning: Methods and Applications (2014, methodology): “Excellent multilingual speech recognition results are reported, far exceeding the earlier results using the GMM–HMM based approaches [225, 420].”
Multilingual acoustic models using distributed deep neural networks (2013, background): “In particular, our main goal is to train more accurate networks for the given data - rather than to bootstrap a language with little or no data from another language by using model adaptation [13, 14], a tandem approach [15, 16, 17, 18, 19], a phone mapping [20], unsupervised pre-training [21], init”
Universal Phone Recognition with a Multilingual Allophone System (2020, background): “The first group, shared phoneme models, creates a shared phoneme inventory of all phonemes from all training languages [3, 4, 5, 6, 7, 10].”
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, methodology): “Bolstering the decoder with a language model (LM) trained on supplemental text data is one such method that improves end-to-end ASR performance [4, 5]; however, more significant gains can be obtained by training on additional synthetically perturbed speech [6, 7, 8], or by multilingual training, whi”
Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers (2013, methodology): “, the IPA universal phone set [15][16]) or by data-driven clustering [15][16][17][18].”
New types of deep neural network learning for speech recognition and related applications: an overview (2013, methodology): “Shallow models such as the GMMs used in the previous generation of acoustic models do not benefit nearly as much as DNNs from being trained on multiple languages simultaneously or from being trained on one language and then modified for another language (e.g. [37]).”
Multilingual MLP features for low-resource LVCSR systems (2012, background): “This common phoneset can be derived either in a data driven fashion or using phonetic sets such as the International Phonetic Alphabet (IPA) [3].”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “This has several important applications: despite much work on multilingual speech recognition [Burget et al., 2010, Lin et al., 2009, Heigold et al., 2013, Bourlard et al., 2011, Cho et al., 2018, Toshniwal et al., 2018, Kannan et al., 2019, Li et al., 2019, Pratap et al., 2020b], many deployed…”
Multilingual acoustic modeling for speech recognition based on subspace Gaussian Mixture Models (2010, background): “The usual approach has been to define a common set of universal phone models with appropriate parameter sharing [3] and train it on data from many languages with eventual adaptation on the data from the language of interest.”
Machine Learning Paradigms for Speech Recognition: An Overview (2013, unknown): “are common and useful approaches [233].”
An Investigation of Deep Neural Networks for Multilingual Speech Recognition Training and Adaptation (2017, background): “The result is consistent with previous work [3, 9, 10].”
Multitask Learning of Deep Neural Networks for Low-Resource Speech Recognition (2015, methodology): “…predictions; [38] improves intent classification in goal-oriented human-machine spoken dialog systems which is particularly successful when the amount of labeled training data is limited; in [39], the MTL approach is used to perform multi-label learning in an image annotation application.”
Deep neural network features and semi-supervised training for low resource speech recognition (2013, background): “In these settings, it is possible to take advantage of transcribed data from other languages to build multilingual acoustic models [1, 2].”
Recent advances in deep learning for speech research at Microsoft (2013, background): “and of model adaptation [35] or neural net initialization [49][51].”
Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model (2017, background): “The usual approach has been to define a common set of universal phone models [7–9] with appropriate parameter sharing [6] and train it on data from many languages with eventual adaptation on the data from the language of interest [10–13].”