Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

By Ammar Ahmad Awan, Ching-Hsiang Chu, Hari Subramoni, Dhabaleswar K. D. K. Panda. 2017

51 citations recorded by the source corpus.

Traditionally, MPI runtimes have been designed for clusters with a large number of nodes. However, with the advent of MPI+CUDA applications and dense multi-GPU systems, it has become important to design efficient communication schemes. This coupled with new application workloads brought forward by Deep Learning frameworks like Caffe and Microsoft CNTK pose additional design constraints due to very large message communication of GPU buffers during the training phase. In this context, special-purpose libraries like NCCL have been proposed. In this paper, we propose a pipelined chain (ring) design for the MPI_Bcast collective operation along with an enhanced collective tuning framework in MVAPICH2-GDR that enables efficient intra-/internode multi-GPU communication. We present an in-depth performance landscape for the proposed MPI_Bcast schemes along with a comparative analysis of NCCL Broadcast and NCCL-based MPI_Bcast. The proposed designs for MVAPICH2-GDR enable up to 14X and 16.6X improvement, compared to NCCL-based solutions, for intra- and internode broadcast latency, respectively. In addition, the proposed designs provide up to 7% improvement over NCCL-based solutions for data parallel training of the VGG network on 128 GPUs using Microsoft CNTK. The proposed solutions outperform the recently introduced NCCL2 library for small and medium message sizes and offer comparable/better performance for very large message sizes.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Hardware and Architecture

Citations in the corpus

PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, background): “Current deep learning frameworks rely either solely or predominantly on all-reduce, which is key to why regular SGD scales well with fast communication hardware (cf. Awan et al., 2018; Panda et al., 2019).”
From distributed machine learning to federated learning: a survey (2021, background): “, Message Passing Interface (MPI) [56] or NVIDIA Collective Communications Library (NCCL) [9], are widely exploited for distributed data processing [171].”
Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect (2019, background): “Awan et al. [60] proposed a pipelined chain design for MPI broadcast collective operations on multi-GPU nodes to facilitate various deep learning frameworks.”
A generic communication scheduler for distributed DNN training acceleration (2019, methodology): “, Stale Synchronous Parallel [20] for PS and different all-reduce algorithms [10, 14]; (4) minimizing network flow completion time by using flow control [26] or Coflow scheduling [13].”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “various communication primitives, such as AllReduce, Reduce, Broadcast, and AllGather, to synchronize partial outputs produced by GPUs (Awan et al., 2018).”