Staleness-Aware Async-SGD for Distributed Deep Learning

By Wei Zhang, Suyog Gupta, Xiangru Lian, Ji Liu. 2015

282 citations recorded by the source corpus.

Deep neural networks have been shown to achieve state-of-the-art performance in several machine learning tasks. Stochastic Gradient Descent (SGD) is the preferred optimization algorithm for training these networks and asynchronous SGD (ASGD) has been widely adopted for accelerating the training of large-scale deep networks in a distributed computing environment. However, in practice it is quite challenging to tune the training hyperparameters (such as learning rate) when using ASGD so as achieve convergence and linear speedup, since the stability of the optimization algorithm is strongly influenced by the asynchronous nature of parameter updates. In this paper, we propose a variant of the ASGD algorithm in which the learning rate is modulated according to the gradient staleness and provide theoretical guarantees for convergence of this algorithm. Experimental verification is performed on commonly-used image classification benchmarks: CIFAR10 and Imagenet to demonstrate the superior effectiveness of the proposed approach, compared to SSGD (Synchronous SGD) and the conventional ASGD algorithm.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Adaptive Task Allocation for Asynchronous Federated Mobile Edge Learning (2019, methodology): “It has been shown in the literature that the loss function of SGDbased ML is minimized (and thus the learning accuracy is maximized) by minimizing the staleness between the gradients in Asynchronous SGD [10].”
Sparse Communication for Distributed Gradient Descent (2017, positive): “Zhang et al. (2015) down-weight gradients based on stale parameters.”
Collaborative Deep Learning in Fixed Topology Networks (2017, background): “Natural Gradient Stochastic Gradient Descent (NG-SGD) based on model averaging [14] and staleness-aware async-SGD [15] have also been developed for distributed deep learning.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “to be Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260] Nondeterministic Comm.[54,121,201] Param”
Training Large Neural Networks with Constant Memory using a New Execution Algorithm (2020, methodology): “Traditional distributed training of neural networks started with data parallelism which keeps a copy of the whole model on each device and partitions data among multiple devices (Dean et al., 2012; Krizhevsky, 2014; Lian et al., 2015; Zhang et al., 2016; Chen et al., 2016a).”
Parle: parallelizing stochastic gradient descent (2017, methodology): “As a means of coupling these variables, consider the loss function of Elastic-SGD (Zhang et al., 2015a): arg min x , x 1 , ..., x n n ∑ a = 1 f ( x a )+ 1 2 ρ x a − x 2 ; (2) where a parameter ρ > 0 couples two replicas x a and x b through a “reference” variable x .”
Distributed stochastic optimization for deep learning (thesis) (2016, unknown): “We have also made an open source project named mpiT on github to facilitate the communication using MPI under Torch, including our implementation of DOWNPOUR and EASGD.”
BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy (2019, background): “…can subscribe the updated weights from a parameter server asynchronously (i.e., removing the synchronization barrier and suppressing bursty trafﬁc) (Niu et al., 2011; Zhang et al., 2015a;b) and decentralized SGD where each learner communicates only with a subset of all learners (Lian et al., 2017).”
GaDei: On Scale-Up Training as a Service for Deep Learning (2016, background): “fference between the timestamp of the learner’s weights and the timestamp of PS’s weights. A large staleness can cause learners to mis-calculate the gradients, which leads to a slower convergence rate[19, 21, 53, 58]. We explain why training with a small mini-batch size can effectively reduce stalen”
The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory (2018, background): “There exists signiﬁcant work on mitigating the effects of asynchrony in applied settings, e.g. [35, 36, 20].”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, background): “W. Zhang et al’s [27] work on n -softsync protocol suggested that instead of waiting for all learners λ to complete calculating gradients as in case of Synchronous SGD, we update weights after collecting gradients from atleast c = learners where n is the hyperparameter controlling staleness of the…”
Adaptive Task Allocation for Mobile Edge Learning (2018, background): “For DL, this is equivalent to maximizing the number of local iterations τ in each global cycle [14].”
MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets (2018, unknown): “In practice, the training task nevertheless works well if the learning rate is adapted in consequence [14], [31], [13].”
Distributed deep learning on edge-devices: Feasibility via adaptive compression (2017, methodology): “The staleness, proportional to the number of workers (please refer to [9], [18] or [29] for in depth phenomenon explanation), is an important issue in asynchronous SGD in general.”
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform (2018, background): “Zhang et al. (Zhang et al., 2016) propose staleness-dependant learning rate to apply stale gradients on weight updates with a decay proportional to the inverse of the weight version difference.”