Faster On-Device Training Using New Federated Momentum Algorithm

By Zhouyuan Huo, Qian Yang, Bin Gu, Lawrence Carin, Heng Huang. 2020

58 citations recorded by the source corpus.

Mobile crowdsensing has gained significant attention in recent years and has become a critical paradigm for emerging Internet of Things applications. The sensing devices continuously generate a significant quantity of data, which provide tremendous opportunities to develop innovative intelligent applications. To utilize these data to train machine learning models while not compromising user privacy, federated learning has become a promising solution. However, there is little understanding of whether federated learning algorithms are guaranteed to converge. We reconsider model averaging in federated learning and formulate it as a gradient-based method with biased gradients. This novel perspective assists analysis of its convergence rate and provides a new direction for more acceleration. We prove for the first time that the federated averaging algorithm is guaranteed to converge for non-convex problems, without imposing additional assumptions. We further propose a novel accelerated federated learning algorithm and provide a convergence guarantee. Simulated federated learning experiments are conducted to train deep neural networks on benchmark datasets, and experimental results show that our proposed method converges faster than previous approaches.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, background): “[26] Zhouyuan Huo, Qian Yang, Bin Gu, Lawrence Carin Huang, et al.”
Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies (2020, background): “[15] Zhouyuan Huo, Qian Yang, Bin Gu, Lawrence Carin, and Heng Huang.”
Federated Learning With Quantized Global Model Updates (2020, background): “…convergence guarantees for FL under various scenarios, considering iid data across the devices [11, 24–27], non-iid data [27, 14, 28, 29], participation of all the devices [30–33], or only a subset of devices at each iteration [34–36, 29, 37], and FL under limited communication constraints [37–40].”
Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data (2020, background): “…performance of FL algorithms on visual classification tasks and show that FedAvg performs worse with increasing nonIIDness than FedMomentum, which can consistently improve the test accuracy (Hsu et al. (2019, 2020)) and theoretically converge to a first-order stationary point (Huo et al. (2020)).”
Personalized Federated Learning using Hypernetworks (2021, background): “This aligns with prior work that highlighted the beneﬁts of local optimization steps in terms of both convergence speed (hence communication cost) and ﬁnal accuracy (McMahan et al., 2017a; Huo et al., 2020).”
Towards Understanding Biased Client Selection in Federated Learning (2022, unknown): “…(Reddi et al., 2021; Haddadpour and Mahdavi, 2019; Khaled et al., 2020; Stich and Karimireddy, 2020; Woodworth et al., 2020; Koloskova et al., 2020; Huo et al., 2020; Zhang et al., 2020; Pathak and Wainwright, 2020; Malinovsky et al., 2020; Sahu et al., 2020) and methods to overcome the adverse…”