Pipelined Back-Propagation for Context-Dependent Deep Neural Networks

By Xie Chen, Adam Eversole, Gang Li, Dong Yu, Frank Torsten Bernd Seide. 2012

90 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Deep Learning: Methods and Applications (2014, background): “, 2012) or over multiple GPUs through pipelined BP (Chen et al., 2012).”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “A pipeline training approach was propoased in [7] and a 3.3x speedup was achieved using 4 GPUs, but this method does not scale beyond number of layers in the neural network.”
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “For (a), we ﬁnd that at any given point in the training, the minibatch size N has an upper limit above which convergence slows notably or fails [19, 8].”
A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion (2013, background): “extraction and recognition, scaling very well from small [8][26][27][28] to medium [3][4][19][36] and to large [2][6][17]”
Multi-GPU Training of ConvNets (2013, background): “Unlike previous work [9, 10, 11], we do not aim to improve the underlying optimization algorithm.”
A scalable approach to using DNN-derived features in GMM-HMM based acoustic modeling for LVCSR (2013, unknown): “, [12]) and large-scale CPU clusters (e.”
Multi-basis adaptive neural network for rapid adaptation in speech recognition (2015, methodology): “It outperforms the conventional Gaussian mixture hidden Markov model (GMM-HMM) system in a variety of large vocabulary continuous speech recognition tasks [1, 2, 3].”
Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks (2012, methodology): “CD-DNN-HMMs have achieved 16% [1][2] and 33% [3][4][5][6] relative recognition error reduction over strong, discriminatively trained CD-GMM-HMMs, respectively, on a voice search (VS) task [10] and the Switchboard (SWB) phone-call transcription task [11].”
Joint training of convolutional and non-convolutional neural networks (2014, background): “Another advantage of the graph structure is that we can split one layer into n parallel parts (striping) [11].”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, background): “Recent works have also explored mitigating these issues when training networks through a combination of data parallelism and PB (Chen et al., 2012; Harlap et al., 2018; Chen et al., 2018; Yang et al., 2019; Zhuang et al., 2019).”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, background): “Chen et al. [6] brieﬂy explore the potential beneﬁts of pipelining minibatches in model parallel training, but do not address the conditions for good statistical eﬃciency, scale, and generality as applicable to large real-world models.”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “[15] briefly explored the potential benefits of pipelining minibatches in model-parallel training, but do not address the conditions for good statistical efficiency, scale, and generality as applicable to large real-world models.”
Stimulated Deep Neural Network for Speech Recognition (2016, background): “In recent years, deep neural networks [1, 2, 3] (DNNs) have successfully been applied to acoustic models of state-of-the-art speech recognition systems.”
Large scale deep neural network acoustic modeling with semi-supervised training data for YouTube video transcription (2013, background): “The largest previous deep network state inventory known to us is 32,000 [18].”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “Strategies to leverage PMP have been explored in previous studies [6, 17, 24, 34].”
On parallelizability of stochastic gradient descent for speech DNNS (2014, background): “We find that at any point during the training, there is an upper limit to N (let’s call it the feasible limit), above which convergence slows notably and eventually fails [11].”
Scalable distributed DNN training using commodity GPU cloud computing (2015, background): “Low single digit speedups are reported in [11], [9] and [12].”