Scaling Distributed Machine Learning with In-Network Aggregation

By Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan R. K. Ports, Peter Richtárik. 2019

553 citations recorded by the source corpus.

Training complex machine learning models in parallel is an increasingly important workload. We accelerate distributed parallel training by designing a communication primitive that uses a programmable switch dataplane to execute a key step of the training process. Our approach, SwitchML, reduces the volume of exchanged data by aggregating the model updates from multiple workers in the network. We co-design the switch processing with the end-host protocols and ML frameworks to provide a robust, efficient solution that speeds up training by up to 300%, and at least by 20% for a number of real-world benchmark models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Natural Compression for Distributed Deep Learning (2019, methodology): “We follow the same communication strategy introduced in SwitchML [40], which aggregates the deep learning model’s gradients using In-Network Aggregation on programmable network switches.”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, methodology): “Some also propose using InfiniBand switch ASIC [28] to accelerate all-reduce, or using P4 switches [58, 59] to accelerate PS.”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, background): “SwitchML and ATP [30, 53] leverages programmable network switches to perform in-network aggregation for cross-rack bandwidth reduction in datacenter environments.”
ATP: In-network Aggregation for Multi-tenant Learning (2021, methodology): “As this improvement only applies to communication, the overall training accel ration ratio depends specifically on the ratio of communication to computation in the DT job [28, 56].”
A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent (2019, background): “, 2014; Konečný and Richtárik, 2018), in-network by a switch (Sapio et al., 2019), or a different technique best suited to the architecture used.”
MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud (2022, background): “It has 200 convolution layers, width factor 8, and its bottleneck block configuration is [6, 8, 46, 6].”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, background): “SwitchML and ATP [27, 47] leverages programmable network switches to perform in-network aggregation for cross-rack bandwidth reduction in datacenter environments.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “RDMA-based OmniReduce outperforms SwitchML* when the sparsity is higher than 60%.”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “However, there exists a fundamental tension between computation and communication [61].”