BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy

By Minsik Cho, Ulrich Finkler, Mauricio J. Serrano, David S. Kung, Hillery C. Hunter. 2019

130 citations recorded by the source corpus.

As deep neural networks get more complex and input datasets get larger, it can take days or even weeks to train a deep neural network to the desired accuracy. Therefore, enabling distributed deep learning at a massive scale is critical, since it offers the potential to reduce the training time from weeks to hours. In this paper, we present BlueConnect, an efficient communication library for distributed deep learning that is highly optimized for popular GPU-based platforms. BlueConnect decomposes a single all-reduce operation into a large number of parallelizable reduce-scatter and all-gather operations to exploit the trade-off between latency and bandwidth, and adapt to a variety of network configurations. Therefore, each individual operation can be mapped to a different network fabric and take advantage of the best performing implementation for the corresponding fabric. According to our experimental results on two system configurations, BlueConnect can outperform the leading industrial communication library by wide margin, and the BlueConnect integrated Caffe2 can significantly reduce synchronization overhead by 87% on 192 GPUs for Resnet-50 training over prior schemes.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “It is also a commonly used primitive with efficient implementation in other types of network topology [37].”
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training (2020, methodology): “choose shard ID based on the network topology, not the other way around. In practice, reduce-scatter and all-gather can be implemented in multiple phases in order to leverage speciﬁc network topology [4]. For instance, for 8 an N M array of devices, reduce-scatter with data size D can be done ﬁrst f”
ATP: In-network Aggregation for Multi-tenant Learning (2021, background): “(2) reducing network traffic – using large batch size to reduce the communication frequency [11, 29, 38]; using quantization [57] or reducing redundancy in SGD [45] to reduce bytes sent over the network; optimizing mixture of local-global aggregation to adapt to network change at runtime [21, 64].”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, background): “BlueConnect is an efficient AllReduce communication library considering heterogeneous networks [9]; unfortunately, it also cannot handle stragglers caused by heterogeneous GPUs.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “Indeed, the fact that communication is a major performance bottleneck in DDL is well-known [32], and many works [10, 35, 39, 44, 58, 66] proposed various optimizations to achieve high-bandwidth collective communication specialized for DDL.”