Distributed Learning over Unreliable Networks

By Hanlin Tang, Chen Yu, Cédric Renggli, S. Kassing, Ankit Singla, Dan Alistarh, Ji Liu, Ce Zhang. 2018

67 citations recorded by the source corpus.

Most of today's distributed machine learning systems assume {\em reliable networks}: whenever two machines exchange information (e.g., gradients or models), the network should guarantee the delivery of the message. At the same time, recent work exhibits the impressive tolerance of machine learning algorithms to errors or noise arising from relaxed communication or synchronization. In this paper, we connect these two trends, and consider the following question: {\em Can we design machine learning systems that are tolerant to network unreliability during training?} With this motivation, we focus on a theoretical problem of independent interest---given a standard distributed parameter server architecture, if every communication between the worker and the server has a non-zero probability $p$ of being dropped, does there exist an algorithm that still converges, and at what speed? The technical contribution of this paper is a novel theoretical analysis proving that distributed learning over unreliable network can achieve comparable convergence rate to centralized or distributed learning over reliable networks. Further, we prove that the influence of the packet drop rate diminishes with the growth of the number of \textcolor{black}{parameter servers}. We map this theoretical result onto a real-world scenario, training deep neural networks over an unreliable network layer, and conduct network simulation to validate the system improvement by allowing the networks to be unreliable.

Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Optimization & Training

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “[427] show that one can achieve convergence rates that are comparable to the case of a reliable network.”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, background): “(2019) have studied time-varying networks; (Yu et al., 2019) in the case of parameter servers.”
MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “Fault-tolerance of NNs [71, 72] allows NNs to adopt approximation with small accuracy loss while improving energy efficiency [62].”
A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “In gradient-update-based FL frameworks [91, 210], as the model gradients are derived from the participant’s private training dataset, the gradients may cause serious privacy leakage [129, 192].”