Multilingual acoustic models using distributed deep neural networks

By Georg Heigold, Vincent Vanhoucke, Andrew W. Senior, Patrick Nguyen, Marc'Aurelio Ranzato, Matthieu Devin, Joseph Dean. 2013

330 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, methodology): “In the context of deep neural networks, MTL has been applied successfully to various problems ranging from language (Liu et al., 2015), to vision (Donahue et al., 2014), and speech (Heigold et al., 2013; Huang et al., 2013).”
TensorFlow: A system for large-scale machine learning (2016, neutral): “, and demonstrate the compelling performance that TensorFlow achieves for several real-world applications. 1 Introduction In recent years, machine learning has driven advances in many different ﬁelds [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76]. We attribute this success to the inventi”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “We and others at Google have performed a wide variety of research using DistBelief including work on unsupervised learning [31], language representation [35, 52], models for image classification and object detection [16, 48], video classification [27], speech recognition [56, 21, 20],”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Multi-lingual training [10, 11], multi-domain training [12, 13], unsupervised pre-training [14, 15], semi-supervised learning [16, 17] and transfer learning [18, 19] are some techniques proposed in the literature to enhance generalization.”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
Multilingual Speech Recognition with a Single End-to-End Model (2017, background): “Most of the previous work on multilingual speech recognition has been limited to making the acoustic model (AM) multilingual [3, 4, 5, 6, 9, 10, 11, 12, 13].”
Unsupervised Pretraining Transfers Well Across Languages (2020, background): “Closer to our work, several works have shared the parameters of a neural network encoder, using feedforward networks [3, 1, 2] or LSTM [11].”
Deep Learning: Methods and Applications (2014, background): “In recent papers of [94, 170] and [150], two research groups independently developed closely related DNN architectures with multi-task learning capabilities for multilingual speech recognition.”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “This has been a long standing interest in the speech community [1, 2, 3, 4, 5, 6, 7] which includes systems able to transcribe multiple languages [8, 9, 10, 11, 12].”
Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “We train the models in a distributed manner using asynchronous stochastic gradient descent (ASGD) optimization technique allowing parallelization of training over a large number of machines on a cluster and enabling large scale training of neural networks [10, 11, 12, 13, 3].”
Sequence-Based Multi-Lingual Low Resource Speech Recognition (2018, positive): “been the so-called “shared hidden layer” model, in which data is passed through a series of shared feed-forward layers, before being separated into multiple language-specific softmax layers, which are trained using cross-entropy [13, 5, 14].”
Design and Learning of Output Representations for Speech Recognition (2013, methodology): “27 In recent years, learning representations using deep models, notably those based on deep neural 28 networks (DNNs), have largely focused on the sensory input data, such as speech and image 29 [20][5][22] with visible successes most notably in industry-scale, real-world speech and image 30 recogni”
Lattice-free State-level Minimum Bayes Risk Training of Acoustic Models (2018, methodology): “In an earlier version of DNN-based ASR, AMs were trained using a frame-level cross-entropy (CE) loss criterion [1, 2, 3], which provided signiﬁ-cant accuracy improvements compared to the Gaussian mixture model (GMM)-based AMs [3, 4, 5].”
Language Independent End-to-End Architecture For Joint Language and Speech Recognition (methodology): “Initial attempts are based on Gaussian mixture model (GMM)/HMM based acoustic model [21, 22], which are extended with DNNs [23] [24] [25].”
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model (2019, background): “Successful strategies for building multilingual acoustic models (AMs) include stacked bottleneck features [1–4], shared hidden layers [5, 6], knowledge distillation [7], and multitask learning [8].”