Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

By Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R. Tallent, K. Barker. IEEE Transactions on Parallel and Distributed Systems, 2019

306 citations recorded by the source corpus.

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

Subfields: Computer Networks and Communications, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/46bb5162da1ad9b3591a9e056550135e7a50bc2b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

Citations in the corpus

Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems (2020, methodology): “node system shown in Figure10. We plot the performance ratio between the execution time predicted by the model for ring and FC topology, ignoring additional software overheads often found in practice [32]. In our model we ﬁx per node bandwidth B, and vary the message size Mand per-node latency , to ”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, methodology): “Correspondingly, even though new innovative memory architectures (e.g., HBM [39]) are proposed, the maximum memory capacity for a single GPU is still limited up to a few tens of gigabytes [16].”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, methodology): “of accessing the buddy memory; this latency is set to be twice that of a local GPU HBM2 access, based on prior characterization work [56], [80].”