Statistical Model Aggregation via Parameter Matching
By Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan H. Greenewald, Trong Nghia Hoang. 2019
38 citations recorded by the source corpus.
We consider the problem of aggregating models learned from sequestered, possibly heterogeneous datasets. Exploiting tools from Bayesian nonparametrics, we develop a general meta-modeling framework that learns shared global latent structures by identifying correspondences among local model parameterizations. Our proposed framework is model-independent and is applicable to a wide range of model types. After verifying our approach on simulated data, we demonstrate its utility in aggregating Gaussian topic models, hierarchical Dirichlet process based hidden Markov models, and sparse Gaussian processes with applications spanning text summarization, motion capture analysis, and temperature forecasting.
Subfields: Artificial Intelligence, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hongyi Wang
- Li Shen
- Dimitris Papailiopoulos
- Mingchen Gao
- Zhenyi Wang
- Qiuling Suo
- Yasaman Khazaeni
- Mikhail Yurochkin
In Signal Processing
- Supriyo Chakraborty
- Dong Yu
- Ambrish Rawat
- Yuekai Sun
- Li Deng
- Jinyu Li
- Yifan Gong
- Frank Torsten Bernd Seide
Citations in the corpus
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, methodology): “We have implemented multiple advanced algorithms including SPAHM [34] which uses a statistical model aggregation via parameter matching for supervised and unsupervised learning, and PFNM [35], which provides a communication efficient method for federated learning of fully-connected networks with…”
- Federated Learning with Matched Averaging (2020, methodology): “The immediate consequence of our result is the extension of PFNM (Yurochkin et al., 2019b) to CNNs. Empirically, see Figure 1, we found that this extension performs well on MNIST with a simpler CNN architecture such as LeNet (LeCun et al., 1998) (4 layers) and significantly outperforms…”
- Meta-learning without data via Wasserstein distributionally-robust model fusion (2022, background): “[Yurochkin et al., 2019b,a] proposes a Bayesian non-parametric framework that considers the neuron matching in a probabilistic manner but only works with simple architecture, e.g., a fully connected network.”
- Scalable inference of topic evolution via models for latent geometric structures (2018, unknown): “2 to perform Federated Learning of neural networks trained on heterogeneous data [31] and proposed a general framework for model fusion [30].”