Distributed deep learning on edge-devices: Feasibility via adaptive compression

By Corentin Hardy, Erwan Le Merrer, Bruno Sericola. 2017

70 citations recorded by the source corpus.

A large portion of data mining and analytic services use modern machine learning techniques, such as deep learning. The state-of-the-art results by deep learning come at the price of an intensive use of computing resources. The leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end servers in datacenters. On the other end, there is a proliferation of personal devices with possibly free CPU cycles; this can enable services to run in users' homes, embedding machine learning operations. In this paper, we ask the following question: Is distributed deep learning computation on WAN connected devices feasible, in spite of the traffic caused by learning tasks? We show that such a setup rises some important challenges, most notably the ingress traffic that the servers hosting the up-to-date model have to sustain. In order to reduce this stress, we propose AdaComp, a novel algorithm for compressing worker updates to the model on the server. Applicable to stochastic gradient descent based approaches, it combines efficient gradient selection and learning rate modulation. We then experiment and measure the impact of compression, device heterogeneity and reliability on the accuracy of learned models, with an emulator platform that embeds TensorFlow into Linux containers. We report a reduction of the total amount of data sent by workers to the server by two order of magnitude (e.g., 191-fold reduction for a convolutional network on the MNIST dataset), when compared to a standard asynchronous stochastic gradient descent, while preserving model accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Adaptive Federated Learning in Resource Constrained Edge Computing Systems (2018, methodology): “entire model parameter vector in every global aggregation step, one can also transmit compressed or quantized model parameters to further save the communication bandwidth, where the compression or quantization can be performed using techniques described in [19] and [20], for instance.”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, background): “However, in FL settings, the synchronous approach, which requires the coordination from a centralised server, has substantial advantages over the asynchronous ones in terms of both communication efficiency and security because it allows advanced technologies to be integrated such as aggregation comp”
3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning (2018, methodology): “the most common non-convex optimization problem used in recent work (Watcharapichat et al., 2016; Alistarh et al., 2017; Hardy et al., 2017; Hsieh et al., 2017; Wen et al., 2017; Zhang et al., 2017; Lin et al., 2018; Wu et al., 2018).”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, unknown): “[438] Distributed deep learning on edge-devices.”
A Survey on End-Edge-Cloud Orchestrated Network Computing Paradigms (2019, methodology): “Following the idea of distributed learning, Hardy et al. [45] propose AdaComp, a distributed deep learning algorithm running on edge devices, which can compress the edge updates to the model on cloud servers.”
MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets (2018, unknown): “This has been quantified by several works [12], [13], and solutions for traffic reduction between workers”