When Edge Meets Learning: Adaptive Control for Resource-Constrained Distributed Machine Learning

By Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin Kwong Leung, Christian Makaya, Ting He, Kevin Sean Chan. 2018

487 citations recorded by the source corpus.

Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to a centralized location. In this paper, we consider the problem of learning model parameters from data distributed across multiple edge nodes, without sending raw data to a centralized place. Our focus is on a generic class of machine learning models that are trained using gradient-descent based approaches. We analyze the convergence rate of distributed gradient descent from a theoretical point of view, based on which we propose a control algorithm that determines the best trade-off between local update and global parameter aggregation to minimize the loss function under a given resource budget. The performance of the proposed algorithm is evaluated via extensive experiments with real datasets, both on a networked prototype system and in a larger-scale simulated environment. The experimentation results show that our proposed approach performs near to the optimum with various machine learning models and different data distributions.

Subfields: Computer Networks and Communications, Environmental Engineering, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Environmental Engineering

In ML Systems & Hardware

Citations in the corpus

Gradient Descent with Compressed Iterates (2019, background): “This situation is common in large-scale distributed optimization involving many edge devices, and common challenges include data heterogeneity [36], privacy [7], resource management [24, 32], and system heterogeneity as well as communication efficiency [15, 17].”
First Analysis of Local GD on Heterogeneous Data (2019, background): “For this reason, almost all prior work assumed bounded dissimilarity [8, 16, 17, 18] and addressed other less challenging aspects of federated learning such as decentralized communication, nonconvexity of the objective or unbalanced data partitioning.”
Federated Machine Learning (2019, background): “In [69], authors considered generic class of machine learning models that are trained using gradient-descent based approaches.”
Federated Learning for Keyword Spotting (2018, background): “Several improvements to the initial FedAvg algorithm have been suggested with a focus on client selection [8], budget-constrained optimization [9] and upload cost reduction for clients [10].”
Adaptive Federated Learning in Resource Constrained Edge Computing Systems (2018, background): “A preliminary version of this work entitled “When edge meets learning: adaptive control for resource-constrained distributed machine learning” was presented at IEEE INFOCOM 2018 [1].”
Scheduling Policies for Federated Learning in Wireless Networks (2019, background): “Particularly, when the amount of training time is limited, solutions are taken by adapting the number of locally computing steps to the variance of global gradient [11], [15], [16], or scheduling the maximum number of UEs in a given time frame [10].”
Adaptive Task Allocation for Asynchronous Federated Mobile Edge Learning (2019, methodology): “The works of [8], [11], [12] aimed to optimize the local number of epochs per node with respect to total global iterations in generic resource-constrained edge environments.”
Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge (2018, background): “The other one is [12], which optimized hyper-parameters of FL (i.”
Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning (2020, background): “In [18]–[21], all devices are involved in each round, which is hardly feasible in practical wireless FL applications due to the limited wireless bandwidth.”
AI-Enhanced Offloading in Edge Computing: When Machine Learning Meets Industrial IoT (2019, methodology): “Existing works mostly discuss how to deploy (compress and reduce) neural networks on edge servers, ignoring the impact of AI-enabled service on offloading [3].”
FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data (2020, background): “Note, that (9) is often used in decentralized optimization to quantify the similarity of local problems [10, 11].”
Task Allocation for Mobile Federated and Offloaded Learning with Energy and Delay Constraints (2020, background): “The merits of our proposed heterogeneity aware (HA) approach are shown by comparing its performances to the heterogeneity-unaware (HU) equal task allocation approach of [4], [11].”
Mobile Edge Intelligence and Computing for the Internet of Vehicles (2019, background): “In [123], built upon the OpenVDAP platform [26], an IoT-enabled public safety service, called AutoVAPS, was proposed.”
Deep Learning With Edge Computing: A Review (2019, background): “[126] further explored this issue by considering some practical concerns with implementation on a real testbed.”
Machine Learning at the Network Edge: A Survey (2019, methodology): “introduced a technique to train machine learning models at the edge without the help of external computation systems such as cloud servers [140].”
All One Needs to Know about Fog Computing and Related Edge Computing Paradigms: A Complete Survey (2018, unknown): “Applications [20, 51, 61, 100, 154, 159, 174, 175, 190, 191, 247, 263, 271, 276, 287, 306, 320, 321, 323, 330–379]”
Client-Edge-Cloud Hierarchical Federated Learning (2019, methodology): “, HierFAVG retrogrades to the FAVG algorihtm. In this case, [p] is the same as fqg, p(t) = 1, ˝ 1˝ 2 = ˝ 1, and G c(t) = h(t (q 1)˝ 1;+ ; q). This is consistent with the result of cloud-based FL in [3]. When ˝ 1 = ˝ 2 = 1, HierFAVG retrogrades to the traditional distributed gradient descent. In this”
SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low Overhead (2019, background): “the data privacy requirements by end users [18].”
A Survey of Incentive Mechanism Design for Federated Learning (2021, unknown): “Wang et al. [7] propose an adaptive local epochs control in each training round due to the clients are resource-constrained.”