Accelerating Federated Learning via Momentum Gradient Descent

By Li Chen, Yunfei Chen, Wenyi Zhang. 2019

346 citations recorded by the source corpus.

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Distilled One-Shot Federated Learning (2020, background): “Server momentum introduces a global momentum parameter that improves convergence theoretically and experimentally [19].”
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning (2019, neutral): “Lastly, to reduce the number of communication rounds, convergence speed can be accelerated via collaboratively adjusting the training momentum [21], [22].”
Federated Learning for Vehicular Internet of Things: Recent Advances and Open Issues (2020, background): “[10] employ momentum gradient descent (MGD), a deterministic gradient descent (DGD) ap-”
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, methodology): “For example, using C-ADMM, each worker needs to ﬁnd the transmission power that achieves the transmission rate R = (32 d/ 1 ms ) bits/sec.”
Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, neutral): “Furthermore, speeding up the convergence is also a useful method, such as those in [210], [217].”
Federated Learning for Internet of Things: A Comprehensive Survey (2021, methodology): “To improve the convergence rate of FL algorithms, a new FL design called Momentum federated learning is presented in [196].”
Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, methodology): “The momentum method adopted by many approaches [67], [68] offers a great potential toward this objective.”
A Framework for Evaluating Gradient Leakage Attacks in Federated Learning (2020, background): “The server iteratively aggregates the local gradients from all Kt clients into the global model, and check if the convergence condition of FL task is met and if not, it starts the next iteration round [23,24,40,41].”
Server Averaging for Federated Learning (2021, background): “On the simple end, Momentum Federated Learning averages the momenta of different devices into a global momentum which is distributed at the start of each round [5].”