Decentralized gradient methods: does topology matter?

By Giovanni Neglia, Chuan Xu, D. Towsley, Gianmarco Calbi. 2020

67 citations recorded by the source corpus.

Consensus-based distributed optimization methods have recently been advocated as alternatives to parameter server and ring all-reduce paradigms for large scale training of machine learning models. In this case, each worker maintains a local estimate of the optimal parameter vector and iteratively updates it by averaging the estimates obtained from its neighbors, and applying a correction on the basis of its local dataset. While theoretical results suggest that worker communication topology should have strong impact on the number of epochs needed to converge, previous experiments have shown the opposite conclusion. This paper sheds lights on this apparent contradiction and show how sparse topologies can lead to faster convergence even in the absence of communication delays.

Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Privacy Amplification by Decentralization (2020, background): “…along edges of a network graph, see e.g. (Lian et al., 2017; Colin et al., 2016; Vanhaesebrouck et al., 2017; Lian et al., 2018; Tang et al., 2018; Bellet et al., 2018; Neglia et al., 2020; Koloskova et al., 2020) for recent work and (Kairouz et al., 2019, Section 2.1 therein) for an overview.”
Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “For these reasons, they have attracted significant interest recently in the machine learning community [41, 63, 42, 62, 3, 51, 46, 31].”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “Moreover, a communication network topology design is also critical to improve the communication efficiency [151], for which a group alternating direction method of multipliers [152] was proposed to form a connectivity chain by dividing the workers into head and tail workers.”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “3] extends some of the conclusions in [79] to dynamic topologies and multiple local updates.”