Optimization of Collective Communication Operations in MPICH

By R. Thakur, Rolf Rabenseifner, William D. Gropp. 2005

1,039 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “aining with a minibatch of 512 or 1024, in order to converge to the best result. Since all-reduce is critical to the scalability of our training, we wrote our own implementation of the ring algorithm [48, 65] for higher performance and better stability. Our implementation avoids extraneous copies be”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “Goyal et al. [14] uses more eﬃcient implementations of all_reduce , like the recursive halving-and-doubling algorithm and the bucket algorithm to reduce the amount of data being sent over the network [39].”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, background): “This primitive is called the collective communication operator in HPC literature (Thakur et al., 2005).”
Distributed Learning over Unreliable Networks (2018, background): “One communication primitive in centralized training is to average/aggregate all models, which is called a collective communication operator in HPC literature (Thakur et al., 2005).”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “More precisely, we use general all-reduce, e.g., recursive halving and doubling algorithm (Thakur et al., 2005; Rabenseifner, 2004), for gradient aggregation among K computation device.”
How to scale distributed deep learning? (2016, background): “Although the latency term scales with O(log(p)), there are fast ring algorithms which have bandwidth term independent of p [11].”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, methodology): “In a cluster with p GPUs, the ring-based all-reduce algorithm splits the data on each GPU into p chunks and do the reduction in p−1 iterations [20].”
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, methodology): “Intuitively, this is because allreduce algorithms use binomial reduction tree and/or butterfly communication patterns internally [27].”
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training (2020, background): “e. In a classic algorithm of reduce-scatter and all-gather on N replicas, the data is partitioned into N pieces, and replicas form a logical ring and exchange pieces with neighbors in multiple rounds [22]. In our fusion implementation, the boundaries of these pieces must exactly match the sharding f”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “Adopting the Ring-AllReduce model of [47], we obtain the total runtime of Pipe-SGD with sequential gradient communication under the limited resource assumption via:”
Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs (2019, background): “In particular, the all-reduce on N nodes have the scaling factor of OðlogNÞ or OðNÞ depending on the network topology [32], [33], [34], [35].”
Variance-based Gradient Compression for Efficient Distributed Deep Learning (2018, background): “In such cases, a type of collective communication algorithms called the ring algorithm is efﬁcient (Thakur et al., 2005).”
BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy (2019, methodology): “We implemented BlueConnect ( BLC ) for GPU in C ++ based on CUDA-aware MPI (IBM, 2017b) and NCCL ver.”
Gemini: A Computation-Centric Distributed Graph Processing System (2016, neutral): “In addition, two helper threads per node are created for inter-node message sending/receiving operations via MPI.”
PipeDream: generalized pipeline parallelism for DNN training (2019, methodology): “Others have explored techniques from the HPC literature to reduce the overhead of communication [9, 24, 50, 51], often using highly specialized networking hardware.”
ImageNet Training in Minutes (2017, background): “In particular, the all-reduce on N nodes have the scaling factor of O(logN ) or O(N ) depending on the network topology [25, 26, 30, 31].”
Integrated Model and Data Parallelism in Training Neural Networks (2017, methodology): “The communication complexity for the data parallel approach using ring algorithm for all-reduce [19] is:”