Federated Learning Based on Dynamic Regularization

By Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, Venkatesh Saligrama. 2021

1,176 citations recorded by the source corpus.

We propose a novel federated learning method for distributively training neural network models, where the server orchestrates cooperation between a subset of randomly chosen devices in each round. We view Federated Learning problem primarily from a communication perspective and allow more device level computations to save transmission costs. We point out a fundamental dilemma, in that the minima of the local-device level empirical loss are inconsistent with those of the global empirical loss. Different from recent prior works, that either attempt inexact minimization or utilize devices for parallelizing gradient computation, we propose a dynamic regularizer for each device at each round, so that in the limit the global and device solutions are aligned. We demonstrate both through empirical results on real and synthetic data as well as analytical results that our scheme leads to efficient training, in both convex and non-convex settings, while being fully agnostic to device heterogeneity and robust to large number of devices, partial participation and unbalanced data.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jiashi Feng
3. Jian Liang
4. Yifan Zhang
5. Dapeng Hu
6. Bingsheng He
7. Qinbin Li
8. Jianhua Lu

In Computer Networks and Communications

1. Yuanming Shi
2. Jianmin Lu
3. Jianhua Lu
4. Dawn Song
5. Dacheng Tao
6. Bingsheng He
7. Khaled Ben Letaief
8. Huan Fu

Citations in the corpus

• Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, neutral): “To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30].”
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “FedDyn [10] dynamically changes the local objectives at each communication round to ensure that the local optimum is asymptotically consistent with the stationary points of the global objective.”
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, background): “…various effective and personalized models have been proposed to rectify the original model (1), including regularizing local loss functions at each device [134], [135], [136], distributionally robust modeling [137], [138], multi-task learning [139], as well as the meta-learning approaches [140].”
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning (2022, methodology): “, FedProx [23], SCAFFOLD [18], FedDyn [1], MOON [22] constrain the direction of local model update to align the local and global optimization objectives.”
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction (2022, methodology): “For the Synthetic dataset, following the setting in [1], we generate three types of data settings, including homogeneity setting which is denoted as ”Synthetic(0,0)”, objective heterogeneity setting which is denoted as ”Synthetic(1,0)”, data heterogeneity setting which denoted as ”Synthetic(0,1)”.”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “Recently, two studies (Li et al., 2018; Acar et al., 2021) penalized local models that have a large divergence from the global model, adding a regularization term to the local optimization process and allowing the global model to converge more reliably.”
• Federated Learning on Non-IID Data Silos: An Experimental Study (2021, methodology): “When preparing this paper, there are other contemporary works [2], [40], [48], [73] on federated learning under non-IID setting.”
• Model-Contrastive Federated Learning (2021, background): “We also notice that there are other related contemporary work [1, 30, 43] when preparing this paper.”