Peer-to-peer Federated Learning on Graphs

By Anusha Lalitha, Osman Cihan Kilinc, Tara Javidi, Farinaz Koushanfar. 2019

228 citations recorded by the source corpus.

We consider the problem of training a machine learning model over a network of nodes in a fully decentralized framework. The nodes take a Bayesian-like approach via the introduction of a belief over the model parameter space. We propose a distributed learning algorithm in which nodes update their belief by aggregate information from their one-hop neighbors to learn a model that best fits the observations over the entire network. In addition, we also obtain sufficient conditions to ensure that the probability of error is small for every node in the network. We discuss approximations required for applying this algorithm to train Deep Neural Networks (DNNs). Experiments on training linear regression model and on training a DNN show that the proposed learning rule algorithm provides a significant improvement in the accuracy compared to the case where nodes learn without cooperation.

Subfields: Artificial Intelligence, Computer Networks and Communications, Graph Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Graph Learning

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “While multi-agent optimization has a long history in the control community, fully decentralized variants of SGD and other optimization algorithms have recently been considered in machine learning both for improved scalability in datacenters [30] as well as for decentralized networks of devices [110,”
Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “More recently, decentralized federated learning architectures without a central server have been considered for peer-topeer learning on graph topologies [15] and in the context of social networks [16].”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, background): “…several works have considered communication-eﬃcient collaborative learning where there is no master node, and the computing nodes learn a model collaboratively in a decentralized manner [Doan et al., 2018, Koloskova et al., 2019, Lalitha et al., 2019, Reisizadeh et al., 2019a, Zhang et al., 2018].”
From distributed machine learning to federated learning: a survey (2021, background): “According to the order to conduct neighborhood averaging and gradient descent, D-SGD has two common types of realizations: average with communication (AWC) [88,96] and average before communication (ABC) [27,178].”
The future of digital health with federated learning (2020, background): “An example of this is peer-to-peer FL, where connections exist between all or a subset of the participants and model updates are shared only between directly-connected sites [37, 38].”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “A current trend [104, 18, 100, 95, 7, 49, 53] is then to replace communication with the orchestrator by peer-to-peer communications between individual silos, which perform local partial aggregations of model updates.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “, multitask learning) [53], robustness guarantees [54], [55], and training over dynamic topologies [56].”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “There are relevant designations in the recent literature, such as peer-to-peer FL [49], server-free FL [50], serverless FL [51], device-to-device FL [52], swarm learning [53], etc. Fig.”
Federated Learning for Healthcare Informatics (2019, background): “Therefore, some researches began to study fully decentralized framework where the central server is not required [41, 60, 85, 91].”
Decentralized Federated Averaging (2021, background): “The papers [5], [6] consider training an ML model with a connected overlay network whose nodes take a Bayesian-like approach by introducing a prior of the parameter space.”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, background): “[48] propose a Bayesian-based distributed algorithm, in which each device updates its belief by aggregating information from its one-hop neighbors to train a model that best fits the observations over the entire network.”
Federated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks (2019, methodology): “Finally, [21][22] propose a peer-to-peer Bayesian-like approach to iteratively estimate the”