A generic communication scheduler for distributed DNN training acceleration

By Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, Chuanxiong Guo. 2019

407 citations recorded by the source corpus.

We present ByteScheduler, a generic communication scheduler for distributed DNN training acceleration. ByteScheduler is based on our principled analysis that partitioning and rearranging the tensor transmissions can result in optimal results in theory and good performance in real-world even with scheduling overhead. To make ByteScheduler work generally for various DNN training frameworks, we introduce a unified abstraction and a Dependency Proxy mechanism to enable communication scheduling without breaking the original dependencies in framework engines. We further introduce a Bayesian Optimization approach to auto-tune tensor partition size and other parameters for different training models under various networking conditions. ByteScheduler now supports TensorFlow, PyTorch, and MXNet without modifying their source code, and works well with both Parameter Server (PS) and all-reduce architectures for gradient synchronization, using either TCP or RDMA. Our experiments show that ByteScheduler accelerates training with all experimented system configurations and DNN models, by up to 196% (or 2.96X of original speed).

Subfields: Computer Vision and Pattern Recognition, Hardware and Architecture, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Neurology

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “General distributed training libraries in PyTorch [67], TensorFlow [68], MXNet [69], and distributed training-specialized libraries such as Horovod [70] and BytePS [71] are designed for distributed training in data centers.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, background): “In general, PS is a distributed key-value store, which can be used for exchanging blocks of model parameters between the workers and the servers (Peng et al., 2019).”
Biscotti: A Blockchain System for Private and Secure Federated Learning (2021, background): “This can be achieved through transfer learning [76] on a restricted parameter space or by better partitioning and rearranging of the tensor updates [77].”
ATP: In-network Aggregation for Multi-tenant Learning (2021, background): “Prior efforts propose to improve gradient aggregation time by (1) smarter network scheduling – increasing the overlap between GPU/CPU computation and network transmission via fine-grained tensors transmission scheduling (per layer instead of the whole gradient or parameter) [32, 37, 52, 70]; combini”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, background): “BytePS and ByteScheduler [24, 49] harnesses idle CPU and network resources and better communication scheduling to improve parameter exchange efficiency.”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, methodology): “We incorporated the idea of tensor partition and pipelining from prior work [34, 55].”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, background): “BytePS and ByteScheduler [20, 43] harnesses idle CPU and network resources and better communication scheduling to improve parameter exchange efficiency.”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “Afewsoftware approaches have been recently proposed to optimize the computation-communication pipeline, ranging from priority-based gradient scheduling and partitioning [56] to advanced synchronization architectures [30, 64].”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (positive): “BytePS [26,43] uniﬁes all-reduce and parameter servers and utilizes heterogeneous resources in data center clusters.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, background): “However, in reality, communication time increases with the number of workers due to scaling overheads [35, 50, 64].”
DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs (2020, methodology): “System architectures [18]–[20] proposed for training neural networks for computer vision and natural language processing are not directly applicable because one critical bottleneck in GNN training is the network traffic of”
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, methodology): “The most relevant ones are PipeDream [8], ByteScheduler [9] and Poseidon [40].”
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, neutral): “Peng et al., 2019; Jiang et al., 2020).”