GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training

By T. Paine, Hailin Jin, Jianchao Yang, Zhe L. Lin, Thomas S. Huang. 2013

100 citations recorded by the source corpus.

The ability to train large-scale neural networks has resulted in state-of-the-art performance in many areas of computer vision. These results have largely come from computational break throughs of two forms: model parallelism, e.g. GPU accelerated training, which has seen quick adoption in computer vision circles, and data parallelism, e.g. A-SGD, whose large scale has been used mostly in industry. We report early experiments with a system that makes use of both model parallelism and data parallelism, we call GPU A-SGD. We show using GPU A-SGD it is possible to speed up training of large convolutional neural networks useful for computer vision. We believe GPU A-SGD will make it possible to train larger networks on larger training sets in a reasonable amount of time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

One weird trick for parallelizing convolutional neural networks (2014, methodology): “In [Paine et al., 2013], the authors implement asynchronous SGD [Niu et al., 2011, Dean et al., 2012] on a GPU cluster with fast interconnects and use it to train the convolutional neural net of [Krizhevsky et al., 2012] using model parallelism and data parallelism.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the notable references in a chronological order are Langford et al. (2009); Niu et al. (2011); Agarwal and Duchi (2011); Paine et al. (2013); Li et al. (2014b); Zhang et al. (2014); Keuper and Pfreundt (2015); De and Goldstein (2015); Feyzmahdavian et al. (2015); Lian et al. (2015); Mania…”
Asynchronous Decentralized Parallel Stochastic Gradient Descent (2017, methodology): “The Asynchronous Parallel Stochastic Gradient Descent ( A-PSGD ) [Agarwal and Duchi, 2011, Feyzmah-davian et al., 2016, Paine et al., 2013, Recht et al., 2011] breaks the synchronization in S-PSGD.”
Deep learning with Elastic Averaging SGD (2014, background): “We however emphasize that to the best of our knowledge the problem of stochastic optimization in the parallel deep learning setting under the communication constraints was never addressed in the literature before (and thus as well a more general problem when the data is distributed).”
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization (2015, methodology): “…successes when applied to parallel stochastic gradient [Niu et al., 2011, Agarwal and Duchi, 2011, Zhang et al., 2014, Feyzmahdavian et al., 2015, Paine et al., 2013], stochastic coordinate descent [Avron et al., 2014, Liu et al., 2014a], dual stochastic coordinate ascent [Tran et al., 2015],…”
Recent advances in convolutional neural networks (2015, methodology): “[68] basically combines asynchronous SGD with GPUs to accelerate the training time by several times compared to training on a single machine.”
Deep Image: Scaling up Image Recognition (2015, neutral): “Similar to this work but at a much smaller scale is the work of Paine et al. (2013), in which an asynchronous SGD algorithm based on a GPU cluster was implemented.”
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order (2016, methodology): “This paper provides a comprehensive and generic analysis to study the speedup property for a broad range of asynchronous parallel stochastic algorithms from the zeroth order to the first order methods.”
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? (2018, methodology): “commensurate surge of interest in the asynchronous parallelization of first-order methods such as (stochastic) gradient descent (Agarwal & Duchi, 2011; Recht et al., 2011; Paine et al., 2013; Chaturapruek et al., 2015; Lian et al., 2015; Feyzmahdavian et al., 2016; Mania et al., 2017), coordinate de”
Theano-based Large-Scale Visual Recognition with Multiple GPUs (2014, background): “Many multi-GPU frameworks has been proposed and implemented (Yadan et al., 2013; Zou et al., 2014; Paine et al., 2013; Krizhevsky, 2014), usually adopting a mixed data and model parallelism.”
Distributed stochastic optimization for deep learning (thesis) (2016, unknown): “We will focus on the problem of reducing the parameter communication overhead between the master and the local workers [50, 16, 60, 43, 48].”
Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review (2017, methodology): “ix multiplication methods [160] have been used to improve the computation speed of CNNs, but there is still enormous room for improvement in the training algorithms of deep CNNs using variants of SGD [161] and their parallelised and distributed implementations. The new algorithms to come are expecte”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “A considerable amount of effort (Zinkevich et al., 2010; Recht et al., 2011; Dean et al., 2012; Zhuang et al., 2013; Paine et al., 2013; Yadan, Adams, Taigman, & Ranzato, 2014; Krizhevsky, 2014) has gone in to parallelizing the training of DCNNs via model parallelism, which entails the use of GPUs,…”