Revisiting Distributed Synchronous SGD

By Jianmin Chen, Rajat Monga, Samy Bengio, Rafal Józefowicz. 2016

850 citations recorded by the source corpus.

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.

Subfields: Computer Networks and Communications, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Optimization & Training

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, methodology): “For simplicity, we consider synchronized algorithms for federated learning [3], where a typical round consists of the following steps: 1.”
Scaling Neural Machine Translation (2018, methodology): “We rely on the second scheme and follow synchronous SGD, which has recently been deemed more efﬁcient than asynchronous SGD (Chen et al., 2016).”
TensorFlow: A system for large-scale machine learning (2016, positive): “We have also built various coordination protocols, and achieved encouraging results with synchronous replication, echoing recent results [11, 19] that contradict the commonly held belief that asynchronous replication is required for scalable learning [14, 21, 46].”
Large Batch Training of Convolutional Networks (2017, methodology): “For example Chen et al. (2016) trained the Inception model with batch B=6400, and Li (2017) trained Resnet-152 for B=5K. The main obstacle for scaling up batch is the instability of training with high LR. Hoffer et al. (2017) tried to use less aggressive "square root scaling" of LR with special form”
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, methodology): “[5] presented a comparison of numerous distributed SGD variants, and although their work also employed the linear scaling rule, it did not establish a small minibatch baseline (the most related result is in v1 of [5] which reported a 0.4% increase of error when the minibatch size increases from…”
Multi-task Self-Supervised Visual Learning (2017, background): “The other approach is synchronous training, where the parameter server accumulates gradients from all workers, applies the accumulated update while all workers wait, and then sends back identical parameters to all workers [4], preventing stale gradients.”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, methodology): “We employed synchronous SGD [4] with momentum of 0.9.”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “A number of recent large scale neural network methods including (Krizhevsky, 2014; Chen et al., 2016; Goyal et al., 2017) use the “linear scaling rule” to accelerate training using parallel computation.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “IMPALA use synchronised parameter update which is vital to maintain data efficiency when scaling to many machines (Chen et al., 2016).”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, positive): “Our experimental results corroborate recent ﬁndings that show that ASP does not reduce end-to-end DNN training time [9, 1, 4].”
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (2017, neutral): “The training used 50 synchronized workers [3].”
How To Backdoor Federated Learning (2018, background): “In comparison to synchronous distributed SGD [6], federated learning reduces the number of participants per round and converges faster.”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “1 INTRODUCTION Distributed optimization (Chen et al., 2016; Goyal et al., 2017) is essential for training large models on large datasets (Radford et al.”
Split learning for health: Distributed deep learning without sharing raw patient data (2018, methodology): “In addition to splitNN [32], techniques of federated deep learning [1] and large batch synchronous stochastic gradient descent (SGD)[19] are currently available approaches for distributed deep learning.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “[17] demonstrates that synchronous distributed SGD offers improved performance for deep learning workloads and is able to alleviate the staleness impact of asynchronous SGD.”
Distributed learning of deep neural network over multiple agents (2018, methodology): “We compare our method against the modern state-of-the-art methods including large-batch global SGD [37] and federated averaging approaches [38].”