A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

By Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, Chuanxiong Guo. 2020

410 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

ATP: In-network Aggregation for Multi-tenant Learning (2021, methodology): “This design [39, 51, 62] as shown in Figure 1 enables data-parallel training, where training data is partitioned and distributed to workers.”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (unknown): “BytePS [26,43] uniﬁes all-reduce and parameter servers and utilizes heterogeneous resources in data center clusters.”
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models (2022, methodology): “Parameter server [10, 13, 14] is the earliest system to support data parallelism, soon replaced by Horovod [30] using allreduce for better performance.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “Several efforts optimize DDL communication ranging from designing high-performance PS software [43] and transfer scheduler [20, 25, 50], to improving collective communication in heterogeneous networks fabrics [10, 28] and within multi-GPU servers [66], to developing in-network reduction systems [35,”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “MLP layers (both top and bottom) are replicated across all trainers (GPUs in figure) in a data-parallel manner, while EMBs are sharded across trainers to exploit model-parallelism [17, 18, 31, 44].”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, background): “BytePS and ByteScheduler [24, 49] harnesses idle CPU and network resources and better communication scheduling to improve parameter exchange efficiency.”
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, background): “The inter-device bandwidth is still one or two orders of magnitude lower than that of data access within a device (Jiang et al., 2020; Narayanan et al., 2019).”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, methodology): “BytePS and ByteScheduler [20, 43] harnesses idle CPU and network resources and better communication scheduling to improve parameter exchange efficiency.”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “However, as shown in Table 1, the latest highly-optimized BytePS [30] and Ring-allreduce [64] only achieve scaling efficiencies(1) of 0.”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “In this approach, workers compute model updates and send them to parameter servers [45, 56, 64].”