Congestion Control for Large-Scale RDMA Deployments

By Yibo Zhu, Haggai Eran, D. Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, J. Padhye, S. Raindel, M. H. Yahia, Ming Zhang. Computer communication review, 2015

903 citations recorded by the source corpus.

Modern datacenter applications demand high throughput (40Gbps) and ultra-low latency (< 10 μs per hop) from the network, with low CPU overhead. Standard TCP/IP stacks cannot meet these requirements, but Remote Direct Memory Access (RDMA) can. On IP-routed datacenter networks, RDMA is deployed using RoCEv2 protocol, which relies on Priority-based Flow Control (PFC) to enable a drop-free network. However, PFC can lead to poor application performance due to problems like head-of-line blocking and unfairness. To alleviates these problems, we introduce DCQCN, an end-to-end congestion control scheme for RoCEv2. To optimize DCQCN performance, we build a fluid model, and provide guidelines for tuning switch buffer thresholds, and other protocol parameters. Using a 3-tier Clos network testbed, we show that DCQCN dramatically improves throughput and fairness of RoCEv2 RDMA traffic. DCQCN is implemented in Mellanox NICs, and is being deployed in Microsoft's datacenters.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/07367703f587dbc3313cc613289c4330cebe5c8c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

HPCC: high precision congestion control (2019, methodology): “Given that traditional software-based network stacks in hosts can no longer sustain the critical latency and bandwidth requirements [43], offloading network stacks into hardware is an inevitable direction in high-speed networks.”
Efficient Memory Disaggregation with Infiniswap (2017, background): “Although we evaluate INFINISWAP in a small-scale environment, recent results suggest that deploying RDMA (thus INFINISWAP) on large datacenters may indeed be feasible [48, 61, 79].”
TIMELY: RTT-based Congestion Control for the Datacenter (2015, methodology): “We did not implement DCTCP in OS-bypass environment due to NIC ﬁrmware limitations on processing ECN feedback [48].”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, methodology): “We also assume that the full bisection bandwidth can be fully utilized due to the newly introduced RDMA congestion control algorithms, e.g., DCQCN [75].”
RDMA over Commodity Ethernet at Scale (2016, background): “This can cause problems like unfairness and victim flow [42].”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, methodology): “In distributed training, all-to-all communication may lead to congestion and elevated levels of Priority Flow Control (PFC) [18] when employing the default DC-QCN [19] protocol at scale.”
Swift: Delay is Simple and Effective for Congestion Control in the Datacenter (2020, methodology): “Protocols such as DCTCP [1], PFC [49], DCQCN [59] and HPCC [34] use explicit feedback from switches to keep network queues short and RPC completion times low.”
Datacenter RPCs can be General and Fast (2018, methodology): “The ConnectX4 NICs used in CX4 are widely deployed in datacenters at Microsoft and Facebook [68, 2], and its Mellanox Spectrum switches perform similarly to Broadcom’s Trident switches used in these datacenters (i.”
Homa: a receiver-driven low-latency transport protocol using network priorities (2018, background): “, with carefully-engineered rate control schemes [2, 21, 34], reserving bandwidth headroom [3], or even strictly limiting the buffer size to a small value [15].”
Azure Accelerated Networking: SmartNICs in the Public Cloud (2018, methodology): “When used as an accelerator, the network connection (along with an RDMA-like lossless transport layer using DCQCN [21]) allows scaling to workloads such as large DNN models”
SilkRoad: Making Stateful Layer-4 Load Balancing Fast and Cheap Using Switching ASICs (2017, methodology): “Therefore, SLBs become a severe bottleneck for many delay-sensitive applications [23, 42].”
Re-architecting datacenter networks and stacks for low latency and high performance (2017, methodology): “DCQCN and NDP do even better, with a completion time just 1% slower than optimal.”