Decentralized Bayesian Learning over Graphs

By Anusha Lalitha, Xinghan Wang, Osman Cihan Kilinc, Yongxi Lu, Tara Javidi, Farinaz Koushanfar. 2019

27 citations recorded by the source corpus.

We propose a decentralized learning algorithm over a general social network. The algorithm leaves the training data distributed on the mobile devices while utilizing a peer to peer model aggregation method. The proposed algorithm allows agents with local data to learn a shared model explaining the global training data in a decentralized fashion. The proposed algorithm can be viewed as a Bayesian and peer-to-peer variant of federated learning in which each agent keeps a "posterior probability distribution" over a global model parameters. The agent update its "posterior" based on 1) the local training data and 2) the asynchronous communication and model aggregation with their 1-hop neighbors. This Bayesian formulation allows for a systematic treatment of model aggregation over any arbitrary connected graph. Furthermore, it provides strong analytic guarantees on converge in the realizable case as well as a closed form characterization of the rate of convergence. We also show that our methodology can be combined with efficient Bayesian inference techniques to train Bayesian neural networks in a decentralized manner. By empirical studies we show that our theoretical analysis can guide the design of network/social interactions and data partitioning to achieve convergence.

Subfields: Artificial Intelligence, Computer Networks and Communications, Graph Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Graph Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, methodology): “[254] shows that incorporating Bayesian methods allows for model aggregation across non-IID data and heterogeneous platforms.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “As demonstrated in Figure 3, FL has various topology deﬁnitions, such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], and Hierarchical FL [47, 48, 49, 50, 51, 52].”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Some recent works [47, 61] have investigated decentralized training over heterogeneous data with local updating schemes.”