The Role of Network Topology for Distributed Machine Learning

By Giovanni Neglia, Gianmarco Calbi, D. Towsley, Gayane Vardoyan. 2019

59 citations recorded by the source corpus.

Many learning problems are formulated as minimization of some loss function on a training set of examples. Distributed gradient methods on a cluster are often used for this purpose. In this paper, we study how the variability of task execution times at cluster nodes affects the system throughput. In particular, a simple but accurate model allows us to quantity how the time to solve the minimization problem depends on the network of information exchanges among the nodes. Interestingly, we show that, even when communication overhead may be neglected, the clique is not necessarily the most effective topology, as commonly assumed in previous works.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Privacy Amplification by Decentralization (2020, background): “Fully decentralized approaches are usually motivated by efficiency and scalability concerns: while a central coordinator can become a bottleneck when dealing with the large number of participants commonly seen in “cross-device” applications [32], in fully decentralized methods each participant can c”
Decentralized gradient methods: does topology matter? (2020, unknown): “Neglia et al. (2019) have proposed approximate formulas to evaluate the throughput of distributed ML systems for some speciﬁc random distribution of the computation time (uniform, exponential, and Pareto).”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, methodology): “Reference [78] studies how to select the degree of a regular topology when the duration of a communication round is determined by stragglers [44, 55].”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “DFL networks are inspired by various network topologies, as detailed in sources, such as [150], [151], [152], [153], and [154].”
Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “In fact, fully decentralized schemes may provide stronger privacy guarantees [12] and speed-up training as they better use communication resources [41, 46] and reduce the effect of stragglers [50].”