Bayesian Nonparametric Federated Learning of Neural Networks
By Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan H. Greenewald, Trong Nghia Hoang, Yasaman Khazaeni. 2019
912 citations recorded by the source corpus.
In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited. We develop a Bayesian nonparametric framework for federated learning with neural networks. Each data server is assumed to provide local neural network weights, which are modeled through our framework. We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision, data pooling and with as few as a single communication round. We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.
Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Statistics and Probability
- Bingsheng He
- Qinbin Li
- Virginia Smith
- Dimitris Papailiopoulos
- Jiayu Zhou
- Shashank Rajput
- Tianjian Chen
- Hangyu Zhu
In Probabilistic & Bayesian Methods
In Federated & Distributed Learning
Citations in the corpus
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “Following [138] and [2], we use latent Dirichlet allocation (LDA) to partition the dataset according to the number of workers involved in training in each round.”
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, methodology): “…advanced algorithms including SPAHM [34] which uses a statistical model aggregation via parameter matching for supervised and unsupervised learning, and PFNM [35], which provides a communication efficient method for federated learning of fully-connected networks with adaptive global model size.”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Alternatives to simple weighted averaging have also been proposed: PFNM [47] and FedMA [48] proposes using neural matching, and [49] uses optimal transport to achieve model aggregation.”
- Salvaging Federated Learning by Local Adaptation (2020, background): “Alternative aggregation schemes (Yurochkin et al., 2019; Guha et al., 2019; Hsu et al., 2019) for various flavors of federated learning provide neither privacy, nor robustness, and because our focus is on mitigating the damage from privacy and robustness mechanisms, we do not analyze them in this pa”
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (2019, methodology): “Closely related to our work, Yurochkin et al. [2019] use a Dirichlet distribution with concentration parameter 0.5 to synthesize non-identical datasets.”
- SecureBoost: A Lossless Federated Learning Framework (2019, background): “Following the same idea, several attempts have been made to reinvent different machine learning models to the federated setting, including decision tree,(19,20) linear/logistic regression, and neural network.(24,25) All the abovemethods are designed for horizontally partitioned data.”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Similarly, one-shot/few-shot heuristics [44, 45, 134] have recently been proposed for the federated setting, but have yet to be theoretically analyzed or evaluated at scale.”
- Federated Visual Classification with Real-World Data Distribution (2020, neutral): “[35] and Hsu et al.”
- Federated Learning with Matched Averaging (2020, methodology): “The immediate consequence of our result is the extension of PFNM (Yurochkin et al., 2019b) to CNNs. Empirically, see Figure 1, we found that this extension performs well on MNIST with a simpler CNN architecture such as LeNet (LeCun et al., 1998) (4 layers) and significantly outperforms…”
- Model Fusion via Optimal Transport (2019, background): “[19] extend [18] to more realistic networks”
- Federated Class-Incremental Learning (2022, background): “Federated Learning (FL) is a decentralized learning framework that can train a global model by aggregating lo-cal model parameters [24, 44, 51, 53].”
- Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “Federated learning (McMahan et al., 2016; Li et al., 2018; Yurochkin et al., 2019) is a learning paradigm that tackles this issue by communicating the parameters instead of the raw data itself.”
- Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning (2022, methodology): “Similar to existing works [1, 12, 41], we use Dirichlet distribution Dir(β) on label radios to simulate the non-iid data distribution among clients, where a smaller β indicates higher data heterogeneity.”
- Scalable inference of topic evolution via models for latent geometric structures (2018, neutral): “2 to perform Federated Learning of neural networks trained on heterogeneous data [31] and proposed a general framework for model fusion [30].”