On parallelizability of stochastic gradient descent for speech DNNS

By Frank Torsten Bernd Seide, Hao Fu, Jasha Droppo, Gang Li, Dong Yu. 2014

94 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Deep Learning: Methods and Applications (2014, methodology): “To make deep learning techniques scalable to very large training data, theoretically sound parallel learning and optimization algorithms together with novel architectures need to be further developed [31, 39, 49, 69, 181, 322, 356].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “The massively parallel nature of GPUs allows additional optimizations, such as accelerated mini-batch gradient descent [379,380,383,384].”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “Optimality for parallelization of DNN training was analyzed in [3], and based on the analysis, a gradient quantization approach (1-bit SGD) was proposed to minimize communication cost [4].”
Demystifying Parallel and Distributed Deep Learning (2018, background): “ize to 8k samples [83 ], 32k samples 249 , and even 64k[218] without losing considerable accuracy. While the generalization issue still exists (Section3), it is not as severe as claimed in prior works[211]. One bottleneck that hinders scaling of data parallelism, however, is the BN operator, which r”
Communication Quantization for Data-Parallel Training of Deep Neural Networks (2016, unknown): “The chief difficulty in scaling either method, and especially the data-parallel approach, is severe communication/computation imbalance: without care, more time is spent communicating than doing useful work [4].”
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters (2017, background): “CNTK demonstrates little negative impact on convergence in speech domains [25, 24].”
ImageNet Training in Minutes (2017, background): “However, Downpour’s performance on 1,600 cores for a globally connected network is not significantly better than a single GPU (Seide et al. 2014b).”
ImageNet Training in 24 Minutes (2017, background): “Thus, state-of-the-art methods often use data-parallelism (Amodei et al. 2015; Chen et al. 2016; Dean et al. 2012; Seide et al. 2014a).”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, background): “This approximation strategy lacks generality and is effective for limited scenarios; it does not hurt convergence for some speech models [34], but hurts statistical performance due to noisy gradients in many others [9, 1].”
Privacy Preserving Machine Learning with Homomorphic Encryption and Federated Learning (2021, background): “theoretically compared the efficiency of distributed SGD (stochastic gradient descent) training in model and data parallel, and pointed out that increasing the size of minibatch can improve the efficiency of data training [15,16].”
Group sparse regularization for deep neural networks (2016, background): “…is the main bottleneck in most parallel implementations of gradient descent, where agents must forward them to a centralized parameter server [6], [7]; and ﬁnally, the resulting models might not work on low-power or embedded devices due to excessive computational power needed for performing…”
Scalable distributed DNN training using commodity GPU cloud computing (2015, background): “Our first observation, as was also pointed out in [9] (see also [14]), is that many techniques for speeding up SGD can be formulated as variants of delaying weight updates.”
100-epoch ImageNet Training with AlexNet in 24 Minutes (2017, background): “However, Downpour’s performance on 1,600 cores for a globally connected network is not signiﬁcantly better than a single GPU (Seide et al. 2014b).”
Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs (2019, background): “However, Downpour’s performance on 1,600 cores for a globally connected network is not significantly better than a single GPU [13].”
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “To achieve concurrent computation, we break each minibatch in half and exchange sub-gradients of one half-minibatch while computing the sub-gradients of the next half-minibatch, using a model that is outdated byN/2 samples (delayed update [19, 8]).”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “This approximation strategy is effective for limited scenarios but lacks generality; it does not hurt convergence for some speech models [47], but has not been shown to be effective for other types of models.”