Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

By Youjie Li, Mingchao Yu, Songze Li, Amir Salman Avestimehr, Nam Sung Kim, Alexander Gerhard Schwing. 2018

112 citations recorded by the source corpus.

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4x compared to conventional approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “Circular communication and training architectures have been considered previously in the context of distributed stochastic gradient descent on clear (unmasked) gradient updates, to reduce communication load [37] or to model dataheterogeneity [38].”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Another technique is pipelining (Li et al., 2018), which overlaps the computation and the communication to hide the communication overhead.”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “Recent rigorous theoretical analysis (Stich, 2018; Wang and Joshi, 2018; Yu et al., 2018; Lin et al., 2018) shows that FedAvg is a special case of averaging periodic SGD (also called local SGD) which allows nodes to perform local updates and infrequent synchronization between them to communicate…”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “Li et al. [2018] provide a systematic analysis of the decentralized learning pipeline.”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “Some works also investigate methods for parallel backpropgation (Huo et al., 2018; Li et al., 2018).”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “We believe that our work not only offers practical insights for those working on LLM training, but also paves the way for future research in this rapidly evolving field.”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “Among the multiple workers, the parameters are synchronized using parameter servers [28] or AllReduce communications [29, 45].”
Data Movement Is All You Need: A Case Study on Optimizing Transformers (2020, background): “…et al., 2018; Dean et al., 2012; Chilimbi et al., 2014; Shazeer et al., 2018; Buchlovsky et al., 2019), pipeline parallelism (Chen et al., 2012; Li et al., 2018; Narayanan et al., 2019; Huang et al., 2019), microbatching (Oyama et al., 2018), and recompu-tation for memory reduction (Chen et…”
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (2021, unknown): “Example techniques include quantization (Zhang et al., 2017; Wangni et al., 2018), decentralization (Lian et al., 2017; Koloskova* et al., 2020; Li et al., 2018), and asynchronous communication (Zheng et al., 2016; Chaturapruek et al., 2015).”