Hop: Heterogeneity-aware Decentralized Training

By Qinyi Luo, Jinkun Lin, Youwei Zhuo, Xuehai Qian. 2019

61 citations recorded by the source corpus.

Recent work has shown that decentralized algorithms can deliver superior performance over centralized ones in the context of machine learning. The two approaches, with the main difference residing in their distinct communication patterns, are both susceptible to performance degradation in heterogeneous environments. Although vigorous efforts have been devoted to supporting centralized algorithms against heterogeneity, little has been explored in decentralized algorithms regarding this problem. This paper proposes Hop, the first heterogeneity-aware decentralized training protocol. Based on a unique characteristic of decentralized training that we have identified, the iteration gap, we propose a queue-based synchronization mechanism that can efficiently implement backup workers and bounded staleness in the decentralized setting. To cope with deterministic slowdown, we propose skipping iterations so that the effect of slower workers is further mitigated. We build a prototype implementation of Hop on TensorFlow. The experiment results on CNN and SVM show significant speedup over standard decentralized training in heterogeneous settings.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Da Zheng
2. Minjie Wang
3. Jiaao He
4. Nguyen T. Nguyen
5. Giovanni Neglia
6. Young-ri Choi
7. Sam H. Noh
8. Gyeongchan Yun

In Computer Networks and Communications

1. Minjie Wang
2. Jiaao He
3. Giovanni Neglia
4. Othmane Marfoq
5. Richard Vidal
6. Chuan Xu
7. Zheng Zhang
8. Xuehai Qian

Citations in the corpus

• Heterogeneity-Aware Asynchronous Decentralized Training (2019, background): “To fundamentally improve the scalability, the general decentralized training [21, 22, 33, 34, 35, 37, 47, 48] also received intensive research interests.”
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “For a decentralized training system, Hop [31] considers the bounded staleness and backup workers, and uses CIFAR-10 for performance”
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “2) confirm that the number of communication rounds to converge is weakly sensitive to the topology (as already observed in [61, 60, 49, 66] and partially explained in [86, 5, 79]).”
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs (2020, background): “Several recent work focuses on relaxing the synchronization of weights [38], [39] in case some workers run slower than others temporally due to some hardware issues.”