Distilling On-Device Intelligence at the Network Edge

By Jihong Park, Shiqiang Wang, Anis Elgabli, Seungeun Oh, Eunjeong Jeong, Han Cha, Hyesung Kim, Seong-Lyun Kim, Medhi Bennis. 2019

33 citations recorded by the source corpus.

Devices at the edge of wireless networks are the last mile data sources for machine learning (ML). As opposed to traditional ready-made public datasets, these user-generated private datasets reflect the freshest local environments in real time. They are thus indispensable for enabling mission-critical intelligent systems, ranging from fog radio access networks (RANs) to driverless cars and e-Health wearables. This article focuses on how to distill high-quality on-device ML models using fog computing, from such user-generated private data dispersed across wirelessly connected devices. To this end, we introduce communication-efficient and privacy-preserving distributed ML frameworks, termed fog ML (FML), wherein on-device ML models are trained by exchanging model parameters, model outputs, and surrogate data. We then present advanced FML frameworks addressing wireless RAN characteristics, limited on-device resources, and imbalanced data distributions. Our study suggests that the full potential of FML can be reached by co-designing communication and distributed ML operations while accounting for heterogeneous hardware specifications, data characteristics, and user requirements.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Science Applications

Citations in the corpus

Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Jiang et al. proposed an efﬁcient learning technique using pruned models, while Park et al. [107] proposed a strategy to generate a high-quality ML model through on-device model output, parameters, and data aggregation.”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [19, 20, 21, 22], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “In distributed learning, these techniques can also be utilized for sharing proxy samples without revealing raw data samples [148].”
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, methodology): “Federated learning is one representative method that ensures learning through periodically exchanging model parameters across workers rather than sending private data samples (McMahan et al., 2017; Kairouz et al., 2019; Park et al., 2019b).”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, background): “Federated learning, since its inception, has attracted considerable attention due to its capability of distributed model training using data collected by possibly a large number of edge and mobile devices, such as Internet of Things (IoT) gateways and smartphones [24, 26, 20, 13].”