Making Deep Belief Networks effective for large vocabulary continuous speech recognition

By Tara N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran, Petr Fousek, Petr Novák, Abdel-rahman Mohamed. 2011

201 citations recorded by the source corpus.

To date, there has been limited work in applying Deep Belief Networks (DBNs) for acoustic modeling in LVCSR tasks, with past work using standard speech features. However, a typical LVCSR system makes use of both feature and model-space speaker adaptation and discriminative training. This paper explores the performance of DBNs in a state-of-the-art LVCSR system, showing improvements over Multi-Layer Perceptrons (MLPs) and GMM/HMMs across a variety of features on an English Broadcast News task. In addition, we provide a recipe for data parallelization of DBN training, showing that data parallelization can provide linear speed-up in the number of machines, without impacting WER.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Tara N. Sainath
5. Yoshua Bengio
6. Yifan Gong
7. S. Renals
8. Shinji Watanabe

In Signal Processing

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Tara N. Sainath
5. Yifan Gong
6. S. Renals
7. Shinji Watanabe
8. Khe Chai Sim

Citations in the corpus

• Deep Learning: Methods and Applications (2014, background): “The success was quickly extended to large vocabulary speech recognition tasks with hundreds and even thousands of hours of training set and with thousands of tri-phone states, including the Switchboard and Broadcast News databases, and Google’s voice search and YouTube tasks [94, 161, 184, 309, 311,”
• Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks (2013, methodology): “Because the temporal structure of the HMM is maintained, we refer to these models as CD-DNN-HMM acoustic models.”
• A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “The most notable groups include all major industrial speech labs worldwide: Microsoft [155, 156, 168, 169], IBM [125, 141, 175], Google [120, 176], and Baidu.”
• Maximum a posteriori adaptation of network parameters in deep models (2015, background): “Despite the recent outstanding results demonstrated by contextdependent, deep-neural-network based hidden Markov models (CD-DNN-HMMs) in various automatic speech recognition (ASR) tasks and data sets [1, 2, 3], these acoustic models, similarly to conventional context-dependent, Gaussian-mixturemodel”
• Feature space maximum a posteriori linear regression for adaptation of deep neural networks (2014, background): “…successes in adopting context dependent deep neural network hidden Markov models (CD-DNN-HMMs) in automatic speech recognition (ASR) have demonstrated promising performance improvements over the conventional Gaussian mixture model (GMM) HMMs [1] in various tasks and datasets [2, 3, 4, 5, 6, 7].”
• F0 contour prediction with a deep belief network-Gaussian process hybrid model (2013, methodology): “Toward the other end of a continuum, we have fully parametric, model-based systems which use the training data only during the learning stage to adapt the model parameters, and then use the models at run-time to generate prosodic parameters that can be used directly in the speech-generation stage.”
• Learning small-size DNN with output-distribution-based criteria (2014, methodology): “Context-dependent deep neural network hidden Markov model (CD-DNN-HMM) has been shown, by many groups [1][2][3][4][5][6], to outperform the conventional Gaussian mixture model (GMM)-HMMs on many automatic speech recognition (ASR) tasks.”
• Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks (2013, methodology): “The DNN architecture for Broadcast News consists of a 5 layer DNN with 1,024 hidden units per layer, and a softmax layer with 2,220 outputs [8], as visually shown by the left hand side of Fig.”
• Learning filter banks within a deep neural network framework (2013, methodology): “Following a recipe similar to [4], during fine-tuning, after one pass through the data, loss is measured on a held-out set and the learning rate is reduced by a factor of 2 if the held-out loss has not improved sufficiently over the previous iteration.”
• Large-Scale Domain Adaptation via Teacher-Student Learning (2017, background): “The success of deep neural networks [1][1][2][3][4] relies on the availability of a large amount of transcribed data to train millions of model parameters.”
• Unsupervised adaptation with domain separation networks for robust speech recognition (2017, background): “Table 1 shows the WER performance of clean, GRL adapted and DSN adapted DNN acoustic models for ASR.”
• Cycle-Consistent Speech Enhancement (2018, background): “We pre-train a clean DNN acoustic model as in Section 3.2 of [31] using 9137 clean training utterances in CHiME-3 to evaluate the ASR word error rate (WER) performance of the test features enhanced by F .”
• The Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition (2013, methodology): “Subsequent work on Google voice search and YouTube data [7] and on Broadcast News [8], [9] confirmed the effectiveness of the CD-DNN-HMMs for large vocabulary speech recognition.”
• Semi-supervised training of Deep Neural Networks (2013, methodology): “In our work, we search for an optimal self-training strategy for DNN, which is trained in three stages: 1. pre-training [8][9][10] – layer-wise training of Re-stricted Boltzmann Machines (RBM) by Contrastive Divergence algorithm, 2. frame-classification training [11][12] – mini-batch Stochastic…”
• Cracking the cocktail party problem by multi-beam deep attractor network (2017, background): “Prior to the deep learning era [4, 5, 6, 7, 8, 9], several attempts were made.”
• Understanding how Deep Belief Networks perform acoustic modelling (2012, methodology): “Motivated by the good performance of DBNs on the TIMIT corpus, several leading speech research groups have used DBN acoustic models for a variety of LVCSR tasks [5, 6] achieving very competitive performance.”
• Sequence-discriminative training of deep neural networks (2013, background): “While computational constraints limited earlier uses of hybrid systems to estimating scaled likelihoods for monophones using a two layered network [2] and recurrent networks [3], recent years have seen a resurgence in their use [4, 5, 6, 7].”
• Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, methodology): “They are trained with The IBM Attila Speech Recognition Toolkit [29] in the following three steps [30, 31, 32, 33]: (1) layer-wise discriminative pretraining using stochastic gradient and cross-entropy loss, (2) training using stochastic gradient and cross-entropy (XENT) loss, and (3) training using”