Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks

By Tara N. Sainath, Brian Kingsbury, Hagen Soltau, Bhuvana Ramabhadran. 2013

67 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Frank Torsten Bernd Seide
5. Arun Narayanan
6. Deliang Wang
7. Jasha Droppo
8. Jianshu Chen

In Signal Processing

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Arun Narayanan
5. Deliang Wang
6. Frank Torsten Bernd Seide
7. Yifan Gong
8. Jasha Droppo

Citations in the corpus

• Deep Learning: Methods and Applications (2014, background): “The success was quickly extended to large vocabulary speech recognition tasks with hundreds and even thousands of hours of training set and with thousands of tri-phone states, including the Switchboard and Broadcast News databases, and Google’s voice search and YouTube tasks [94, 161, 184, 309, 311,”
• 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “Even when other techniques are used, such as the Hessian-free method [26, 15, 12], BP still constitutes a significant portion of the training time.”
• A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property (2013, background): “…deep neural networks (DNN) that drastically cut errors in large vocabulary speech recognition (Yu et al., 2010; Dahl et al., 2011; Seide et al., 2011; Dahl et al., 2012; Kingsbury et al., 2012; Hinton et al., 2012; Deng et al., 2013b,c; Yu et al., 2013; Dahl et al., 2013; Sainath et al., 2013).”
• Improving Robustness of Deep Neural Network Acoustic Models via Speech Separation and Joint Adaptive Training (2015, background): “Not only has DNN-AMs improved performance in relatively clean conditions [34], it has”
• On parallelizability of stochastic gradient descent for speech DNNS (2014, methodology): “Even when other techniques are used, such as the Hessian-free method [18, 19, 9], BP still often constitutes a significant portion of their training time.”
• Sequence classification using the high-level features extracted from deep neural networks (2014, unknown): “The use of fully-connected feed-forward deep neural networks (DNNs) in the architecture of DNN-HMM (hidden Markov Model) has dramatically reduced speech recognition errors in recent years [4][5][8][9][19][21][22][28][29][36][37].”