Dynamic Adaptive DNN Surgery for Inference Acceleration on the Edge

By Chuang Hu, Wei Bao, Dan Wang, Fengming Liu. IEEE Conference on Computer Communications, 2019

394 citations recorded by the source corpus.

Recent advances in deep neural networks (DNNs) have substantially improved the accuracy and speed of a variety of intelligent applications. Nevertheless, one obstacle is that DNN inference imposes heavy computation burden to end devices, but offloading inference tasks to the cloud causes transmission of a large volume of data. Motivated by the fact that the data size of some intermediate DNN layers is significantly smaller than that of raw input data, we design the DNN surgery, which allows partitioned DNN processed at both the edge and cloud while limiting the data transmission. The challenge is twofold: (1) Network dynamics substantially influence the performance of DNN partition, and (2) State-of-the-art DNNs are characterized by a directed acyclic graph (DAG) rather than a chain so that partition is greatly complicated. In order to solve the issues, we design a Dynamic Adaptive DNN Surgery (DADS) scheme, which optimally partitions the DNN under different network condition. Under the lightly loaded condition, DNN Surgery Light (DSL) is developed, which minimizes the overall delay to process one frame. The minimization problem is equivalent to a min-cut problem so that a globally optimal solution is derived. In the heavily loaded condition, DNN Surgery Heavy (DSH) is developed, with the objective to maximize throughput. However, the problem is NP-hard so that DSH resorts an approximation method to achieve an approximation ratio of 3. Real-world prototype based on self-driving car video dataset is implemented, showing that compared with executing entire the DNN on the edge and cloud, DADS can improve latency up to 6.45 and 8.08 times respectively, and improve throughput up to 8.31 and 14.01 times respectively.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Neurology

View Paper: https://www.semanticscholar.org/paper/a82bebb9c51b7b0a57ac0026f47344b948acc9e0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “on [34], partitions the CNN between a device-mapped head and a cloud-mapped tail and selects a single split point based on the device and cloud load as well as the network conditions. Similarly, DADS [22] tackles CNN offloading, but from a scheduler-centric standpoint, with the aim to yield the opti”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, neutral): “AIGC service provisioning based on edge intelligence is computationally and communication-intensive for resourceconstrained edge servers and mobile devices [132].”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, background): “, knowledge distillation [36], runtime neural pruning [37], and DNN partitioning and offloading [38], do not serve our purpose.”
A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference (2020, background): “Hu et al. (2019) showed a real-world prototype of an optimal model partitioning, which is based on a self-driving car video dataset, can improve latency and throughput of partitioned models on the cloud and edge by 6.5–14 × , respectively.”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “[80] propose an approximation algorithm that provides worst-case performance guarantee, based on the graph min-cut method.”