Asynchronous stochastic gradient descent for DNN training

By Shanshan Zhang, Ce Zhang, Zhao You, Rong Zheng, Bo Xu. 2013

136 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Yoshua Bengio
5. Hui Jiang
6. Jasha Droppo
7. Mohit Bansal
8. Jianfeng Gao

In Signal Processing

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Chin-Hui Lee
5. Frank Torsten Bernd Seide
6. Jasha Droppo
7. Hui Jiang
8. Xiangang Li

Citations in the corpus

• Distributed Learning over Unreliable Networks (2018, neutral): “, 2018), and asynchronous communication (Lian et al., 2017b; Zhang et al., 2013; Lian et al., 2015).”
• Demystifying Parallel and Distributed Deep Learning (2018, neutral): “Distributed Deep Learning: An In-Depth Concurrency Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260] Nondeterministic Comm.[54,121,201] Parameter Distribution and Communicat”
• Multidimensional Residual Learning Based on Recurrent Neural Networks for Acoustic Modeling (2016, methodology): “For computational efficiency, the model is trained on our asynchronous stochastic gradient descent (ASGD) training platform [25], with four GPUs operating in parallel on 12 utterances at a time for each.”
• Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? (2018, methodology): “In this broad landscape, distributed asynchronous stochastic gradient descent (DASGD) is one of the most widely deployed methods for training large-scale machine learning models, particularly when a stochastic gradient of the underlying learning objective is easily computable (Dean et al., 2012a;b; ”
• Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering (2016, methodology): “In [27], ASGD has also been used to parallelize DNN training on multiple GPU cards in a single computing server, which achieves a 3.”
• Scalable distributed DNN training using commodity GPU cloud computing (2015, background): “There have been many notable attempts at scaling up distributed training across compute nodes [13] [8] [11] [4].”
• Rectified linear neural networks with tied-scalar regularization for LVCSR (2015, methodology): “The asynchronous stochastic gradient descent (ASGD) in [12] is used to parallelize the DNN training among multiple GPUs.”
• Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition (2014, unknown): “Besides, in order to train these networks on multi-GPU devices, asynchronous stochastic gradient descent (ASGD) [24][25] is adopted.”
• Modeling speaker variability using long short-term memory networks for speech recognition (2015, methodology): “We implemented the network training on multi-GPU devices with asynchronous stochastic gradient descent algorithm (ASGD) [38][39].”
• An experimental study on joint modeling of mixed-bandwidth data via deep neural networks for robust speech recognition (2016, methodology): “In the training procedure, we aim at minimizing the following loss function with the asynchronous stochastic gradient descent (ASGD) algorithm [20] in a mini-batch mode: where ˆ x n − τ and x n + τ n − τ are the n -th D (2 τ + 1) -dimensional vectors of estimated and reference wideband features,…”
• Multi-GPU Training of ConvNets (2013, background): “Unlike previous work [9, 10, 11], we do not aim to improve the underlying optimization algorithm.”
• Learning Parameters and Constitutive Relationships with Physics Informed Deep Neural Networks (2018, methodology): “We use a quasi-Newton optimizer such as L-BFGS-B instead of stochastic gradient descent [28] (a more common optimizer for DNNs) because of its superior rate of convergence and more favorable computational cost for problems with a relatively small number of observed data.”
• On parallelizability of stochastic gradient descent for speech DNNS (2014, background): “9B-parameter vision network, while [6] reports a 3.”
• 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “Then, Section 3 will introduce the 1-bit quantization approach, and Section 4 the data-parallel SGD system we implemented based on this.”