TicTac: Accelerating Distributed Deep Learning with Communication Scheduling

By Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, Roy H. Campbell. 2018

214 citations recorded by the source corpus.

State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in these communication-heavy systems depends on the computation time, communication time and the extent of overlap of computation and communication. In this work, we identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We develop a system, TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing near-optimal overlap of communication and computation. TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization. Our system is implemented over TensorFlow and requires no changes to the model or developer inputs. TicTac improves the throughput by up to $37.7\%$ in inference and $19.2\%$ in training, while also reducing straggler effect by up to $2.3\times$. Our code is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chuanxiong Guo
2. Feng Yan
3. Yinlong Xu
4. Yonggang Wen
5. Aditya Akella
6. Chuan Wu
7. Ruichuan Chen
8. Jun Yi

In Computer Vision and Pattern Recognition

1. Kshiteej S. Mahajan
2. Yong Cui
3. Chuanxiong Guo
4. Feng Yan
5. Yinlong Xu
6. Yonggang Wen
7. Aditya Akella
8. Chuan Wu

Citations in the corpus

• Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “Other than gradients compression, there are other approaches aiming at addressing the communication bottleneck in data parallel training, such as using RDMA [80], adopting Ringallreduce [7, 8, 29], co-designing gradient synchronization with the physical topology [35, 38], and priority-based scheduli”
• A generic communication scheduler for distributed DNN training acceleration (2019, methodology): “Communication scheduling [18, 21] is a good solution to this problem.”
• OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, unknown): “…suitable for DL models with a relatively small set of parameters (usually less than tens of millions of parameters), where near-linear speed-up can be achieved once back propagation maximally overlaps with gradient/parameter communication (jea, 2021; Hashemi et al., 2019; 1OneFlow Research.”
• Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “Several efforts optimize DDL communication ranging from designing high-performance PS software [43] and transfer scheduler [20, 25, 50], to improving collective communication in heterogeneous networks fabrics [10, 28] and within multi-GPU servers [66], to developing in-network reduction systems [35,”
• ATP: In-network Aggregation for Multi-tenant Learning (2021, background): “Prior efforts propose to improve gradient aggregation time by (1) smarter network scheduling – increasing the overlap between GPU/CPU computation and network transmission via fine-grained tensors transmission scheduling (per layer instead of the whole gradient or parameter) [32, 37, 52, 70]; combini”
• A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, background): “(2) Communication scheduling and pipelining: Recent work explores to better overlap the computation and communication by prioritybased scheduling and tensor partition [31, 34, 55].”
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes (2019, background): “ironments [31], [32]. Overlapping the communication of the DNN’s top layers with the computation of bottom layers could also help to reduce communication time for distributed DNN training [22], [30], [33], [34]. Some distributed DNN systems support to select and transmit important gradients at the l”