Adaptive Federated Learning in Resource Constrained Edge Computing Systems
By Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin Kwong Leung, Christian Makaya, Ting He, Kevin Sean Chan. 2018
2,085 citations recorded by the source corpus.
Emerging technologies and applications including Internet of Things, social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to a centralized location. In this paper, we consider the problem of learning model parameters from data distributed across multiple edge nodes, without sending raw data to a centralized place. Our focus is on a generic class of machine learning models that are trained using gradient-descent-based approaches. We analyze the convergence bound of distributed gradient descent from a theoretical point of view, based on which we propose a control algorithm that determines the best tradeoff between local update and global parameter aggregation to minimize the loss function under a given resource budget. The performance of the proposed algorithm is evaluated via extensive experiments with real datasets, both on a networked prototype system and in a larger-scale simulated environment. The experimentation results show that our proposed approach performs near to the optimum with various machine learning models and different data distributions.
Subfields: Artificial Intelligence, Computer Networks and Communications, Federated & Distributed Learning, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Richtárik
- Jakub Konecný
- Martin Jaggi
- Virginia Smith
- Medhi Bennis
- Sebastian U. Stich
- H. Vincent Poor
- Jianyu Wang
In Computer Networks and Communications
- Medhi Bennis
- Merouane Debbah
- Jihong Park
- Khaled Ben Letaief
- Walid Saad
- H. Vincent Poor
- Anis Elgabli
- Sumudu Samarakoon
In Federated & Distributed Learning
- Jianyu Wang
- Gauri Joshi
- Peter Richtárik
- Virginia Smith
- Jakub Konecný
- Zachary B. Charles
- Tian Li
- Sebastian U. Stich
In ML Systems & Hardware
- Medhi Bennis
- H. Vincent Poor
- Ahmed Khaled
- Peter Richtárik
- Jakub Konecný
- Jihong Park
- Sebastian U. Stich
- Anis Elgabli
Citations in the corpus
- Federated Optimization in Heterogeneous Networks (2018, methodology): “Further, they rely on specific solvers to be used on each device (either SGD or GD), as compared to the solveragnostic framework proposed herein, and add additional assumptions of convexity (Wang et al., 2019) or uniformly bounded gradients (Yu et al.”
- Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, methodology): “The work in [20] obtained a convergence bound for SGD based FL that incorporates non-independent-and-identically-distributed (non-i.”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “[36] Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “This analysis was further expanded to the non-IID data partition and client sampling cases by [11–18, 23, 24].”
- Adaptive Federated Optimization (2020, methodology): “In order to analyze FEDAVG in heterogeneous settings, many works derive convergence rates that depend on the amount of heterogeneity (Li et al., 2018; Wang et al., 2019; Khaled et al., 2019; Li et al., 2019b).”
- SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “Several analyses bound this drift by using a very small step-size and assuming that the local updates admit bounded magnitude (Wang et al., 2019; Yu et al., 2019; Li et al., 2019b).”
- GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, background): “…per communication round, exchanging model updates can be restricted only to the workers whose computation delays are less than a target threshold (Wang et al., 2018), or to the workers whose updates are sufficiently changed from the preceding updates, with respect to gradients (Chen et al., 2018)…”
- On the Convergence of FedAvg on Non-IID Data (2019, methodology): “The prior work (Coppola, 2015; Zhou and Cong, 2017; Stich, 2018; Yu et al., 2019; Wang and Joshi, 2018; Wang et al., 2019) requires the full device participation in the aggregation step of FedAvg.”
- Wireless Network Intelligence at the Edge (2018, background): “, sparsification [192], opportunistic uploading based on gradient magnitudes [193], and adaptively adjusting the communication intervals [31], [194].”
- LEAF: A Benchmark for Federated Settings (2018, background): “…yet do not provide a realistic model of a federated scenario, e.g., artificial partitions of MNIST, MNIST-fashion or CIFAR-10 [22, 13, 8, 3, 11, 32, 34]; (2) realistic but proprietary federated datasets, e.g., data from an unnamed social network in [22], crowdsourced voice commands in [18], and…”
- FedDANE: A Federated Newton-Type Method (2019, methodology): “Other works analyze different methods with non-identically distributed data, but under different (possibly) limiting assumptions, such as using SGD as a specific local solver [6], full device participation [14, 15], convexity [3, 6, 14], or uniformly-bounded gradients [6, 15].”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “ed model will be biased towards devices with favorable network conditions. Theoretically, while several recent works have investigated convergence guarantees of variants of federated learning methods [56, 123, 131, 132], few analyses allow for low participation [e.g., 65, 106], or study directly the”
- Overcoming Forgetting in Federated Learning on Non-IID Data (2019, neutral): “case) [1, 4].”
- Age-Based Scheduling Policy for Federated Learning in Mobile Edge Networks (2019, background): “This salient feature of on-device training brings along great advantages of eliminating the large communication overheads as well as preserving data privacy, and hence making FL particularly relevant for mobile applications [6–10].”
- Scheduling Policies for Federated Learning in Wireless Networks (2019, methodology): “Particularly, when the amount of training time is limited, solutions are taken by adapting the number of locally computing steps to the variance of global gradient [11], [15], [16], or scheduling the maximum number of UEs in a given time frame [10].”
- Adaptive Task Allocation for Asynchronous Federated Mobile Edge Learning (2019, background): “The works of [8], [11], [12] aimed to optimize the local number of epochs per node with respect to total global iterations in generic resource-constrained edge environments.”