Multilingual training of deep neural networks

By Arnab Ghoshal, Pawel Swietojanski, S. Renals. 2013

224 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Shinji Watanabe
5. Tara N. Sainath
6. Yoshua Bengio
7. Yu Zhang
8. James R. Glass

In Natural Language Processing

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Yoshua Bengio
5. Tara N. Sainath
6. Graham Neubig
7. Ralf Schlüter
8. Shinji Watanabe

Citations in the corpus

• Multilingual Speech Recognition with a Single End-to-End Model (2017, background): “Most of the previous work on multilingual speech recognition has been limited to making the acoustic model (AM) multilingual [3, 4, 5, 6, 9, 10, 11, 12, 13].”
• Effectiveness of self-supervised pre-training for speech recognition (2019, methodology): “Our work is inspired by research efforts reducing the dependence on labeled data for building ASR systems through unsupervised unit discovery and acoustic representation leaning [5, 6, 7, 8, 9], and through multi- and cross-lingual transfer learning in low-resource conditions [46, 47, 48, 49, 50, 51”
• Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “This has been a long standing interest in the speech community [1, 2, 3, 4, 5, 6, 7] which includes systems able to transcribe multiple languages [8, 9, 10, 11, 12].”
• Multilingual representations for low resource speech recognition and keyword search (2015, background): “Multilingual (ML) models have been shown to outperform unilingual models for ASR in low resource languages [1–8].”
• An Investigation of Deep Neural Networks for Multilingual Speech Recognition Training and Adaptation (2017, background): “In [7], the authors proposed to train DNNs on a sequence of target languages, progressively swapping the output layer with each new language.”
• Hybrid acoustic models for distant and multichannel large vocabulary speech recognition (2013, methodology): “We call this the multi-style training, and it is related to our previous work [33], where the same basic concept was used to train DNNs in a multilingual fashion.”
• Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
• Low Resource Multi-modal Data Augmentation for End-to-end ASR (2018, methodology): “Bolstering the decoder with a language model (LM) trained on supplemental text data is one such method that improves end-to-end ASR performance [4]; however, more significant gains can be obtained by training on additional synthetically perturbed speech [5, 6, 7], or by multilingual training, which ”
• Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model (2019, background): “Successful strategies for building multilingual acoustic models (AMs) include stacked bottleneck features [1–4], shared hidden layers [5, 6], knowledge distillation [7], and multitask learning [8].”
• Multitask Learning of Deep Neural Networks for Low-Resource Speech Recognition (2015, methodology): “Successful examples include tied-mixture HMM [12], [13], subspace Gaussian mixture model (SGMM) [14], Bayesian sensing HMM [15], and canonical state model [16].”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Most hybrid multilingual works focus on building an acoustic model with shared hidden layers [70, 84, 100], while every language has its own lexicon model and language model.”
• Improving Attention-Based End-to-End ASR Systems with Sequence-Based Loss Functions (2018, methodology): “Inspired by the swapping-output-layer style multi-lingual modeling in speech recognition [31, 32, 33, 34], we also propose to do alternate training instead of using a predetermined weight to interpolate CTC loss and XENT loss.”
• Language Independent End-to-End Architecture For Joint Language and Speech Recognition (methodology): “Initial attempts are based on Gaussian mixture model (GMM)/HMM based acoustic model [21, 22], which are extended with DNNs [23] [24] [25].”
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data (2021, background): “In the supervised case, transfer learning methods learn features on large high-resource datasets and directly use them in similar but data-poor task (Ghoshal et al., 2013).”
• Speech recognition and keyword spotting for low-resource languages: Babel project research at CUED (2014, background): “Alternatively Hybrid systems, with target language specific output layers, have also been investigated [22, 6, 23].”
• Multilingual deep neural network based acoustic modeling for rapid language adaptation (2014, background): “We also found that KL-HMM based decoding consistently outperforms conventional hybrid decoding, especially in low-resource scenarios.”
• Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model (2017, methodology): “Instead of updating only the output layers [11, 14], we find reestimating all the parameters work better.”