DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters

By Zhuoran Zhao, Kamyar Mirzazad Barijough, Andreas Gerstlauer. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018

449 citations recorded by the source corpus.

Edge computing has emerged as a trend to improve scalability, overhead, and privacy by processing large-scale data, e.g., in deep learning applications locally at the source. In IoT networks, edge devices are characterized by tight resource constraints and often dynamic nature of data sources, where existing approaches for deploying Deep/Convolutional Neural Networks (DNNs/CNNs) can only meet IoT constraints when severely reducing accuracy or using a static distribution that cannot adapt to dynamic IoT environments. In this paper, we propose DeepThings, a framework for adaptively distributed execution of CNN-based inference applications on tightly resource-constrained IoT edge clusters. DeepThings employs a scalable Fused Tile Partitioning (FTP) of convolutional layers to minimize memory footprint while exposing parallelism. It further realizes a distributed work stealing approach to enable dynamic workload distribution and balancing at inference runtime. Finally, we employ a novel work scheduling process to improve data reuse and reduce overall execution latency. Results show that our proposed FTP method can reduce memory footprint by more than 68% without sacrificing accuracy. Furthermore, compared to existing work sharing methods, our distributed work stealing and work scheduling improve throughput by <inline-formula> <tex-math notation="LaTeX">$1.7\times -2.2\times$ </tex-math></inline-formula> with multiple dynamic data sources. When combined, DeepThings provides scalable CNN inference speedups of <inline-formula> <tex-math notation="LaTeX">$1.7\times$ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$3.5\times$ </tex-math></inline-formula> on 2–6 edge devices with less than 23 MB memory each.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/e403972034de384a438e737a1b657de8d0de2500

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, methodology): “1) General Methods for Model Optimization: On one hand, increasing the depth and width of DL models with nearly constant computation overhead is one direction of optimization, such as inception [126] and deep residual networks [127] for CNNs.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “Different methods have been developed [207], [208] to partition a pre-trained DNN over several mobile devices in order to accelerate DNN inference on devices.”
CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices (2020, methodology): “CoEdge employs a similar parallel workflow as DeepThings [14],”
Edge Machine Learning for AI-Enabled IoT Devices: A Review (2020, methodology): “In the distributed perspective the DNN computations can be distributed across multiple peer edge devices, like in DeepThings [90].”
Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, unknown): “munication accuracy and weaken the adaptability to dynamic IoE environments when applying EL with conventional DL methods like CNNs [233].”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “[86] Z. Zhao, K. M. Barijough, and A. Gerstlauer, “DeepThings: Distributed adaptive deep learning inference on resource-constrained IoT edge clusters,” IEEE Trans.”
Machine Learning at the Network Edge: A Survey (2019, methodology): “DeepThings is a framework for adapting CNN-based inference applications on resource-constrained devices [153].”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “Simultaneously, various works [50, 51, 85] have examined the case where the client device can offload to other devices in the local network.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “For example, in DeepThings [375], feature maps are partitioned into nonoverlapping tiles using fused tile partitioning (FTP) to assign them to different edge devices.”