Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning

By Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh. 2020

251 citations recorded by the source corpus.

Federated learning is a challenging optimization problem due to the heterogeneity of the data across different clients. Such heterogeneity has been observed to induce client drift and significantly degrade the performance of algorithms designed for this setting. In contrast, centralized learning with centrally collected data does not experience such drift, and has seen great empirical and theoretical progress with innovations such as momentum, adaptivity, etc. In this work, we propose a general framework Mime which mitigates client-drift and adapts arbitrary centralized optimization algorithms (e.g.\ SGD, Adam, etc.) to federated learning. Mime uses a combination of control-variates and server-level statistics (e.g. momentum) at every client-update step to ensure that each local update mimics that of the centralized method. Our thorough theoretical and empirical analyses strongly establish Mime's superiority over other baselines.

Subfields: Artificial Intelligence, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

Citations in the corpus

Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms (2020, methodology): “‡ the best results taken from [Karimireddy et al., 2020].”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “In the context of federated learning, momentum has been widely incorporated and empirically shown to enhance performance (Wang et al., 2020b; Karimireddy et al., 2020a; Khanduri et al., 2021; Das et al., 2022).”
On Large-Cohort Training for Federated Learning (2021, background): “One area pertinent to our work is that of variance reduction for federated learning, which can mitigate negative effects of data heterogeneity [28, 29, 73].”
Learning from History for Byzantine Robust Optimization (2020, background): “To deploy robust learning for real world applications, many other issues such as data heterogeneity become important (Kairouz et al., 2019; Karimireddy et al., 2020b).”