Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices

By Surat Teerapittayanon, Bradley McDanel, H. T. Kung. 2017

800 citations recorded by the source corpus.

We propose distributed deep neural networks (DDNNs) over distributed computing hierarchies, consisting of the cloud, the edge (fog) and end devices. While being able to accommodate inference of a deep neural network (DNN) in the cloud, a DDNN also allows fast and localized inference using shallow portions of the neural network at the edge and end devices. When supported by a scalable distributed computing hierarchy, a DDNN can scale up in neural network size and scale out in geographical span. Due to its distributed nature, DDNNs enhance sensor fusion, system fault tolerance and data privacy for DNN applications. In implementing a DDNN, we map sections of a DNN onto a distributed computing hierarchy. By jointly training these sections, we minimize communication and resource usage for devices and maximize usefulness of extracted features which are utilized in the cloud. The resulting system has built-in support for automatic sensor fusion and fault tolerance. As a proof of concept, we show a DDNN can exploit geographical diversity of sensors to improve object recognition accuracy and reduce communication cost. In our experiment, compared with the traditional method of offloading raw sensor data to be processed in the cloud, DDNN locally processes most sensor data on end devices while achieving high accuracy and is able to reduce the communication cost by a factor of over 20x.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In ML Systems & Hardware

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “This strategy was popularized by BranchyNet [63, 64].”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, methodology): “These methods include the scalable-effort classifier [32], the conditional deep learning classifier [19], the distributed neural network [31], the edge-host partitioned neural”
Survey of Attacks and Defenses on Edge-Deployed Neural Networks (2019, background): “For example, in [13], the authors show that in convolutional neural networks (CNN), processing a few of the first convolutional layers before sending the data to the cloud achieves higher power savings compared to processing the whole network on”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “[433] Neural network on cloud and end devices.”
Differentiable Branching In Deep Networks for Fast Inference (2020, background): “The successful application of deep networks in energy-constrained applicative scenarios, such as IoT, embedded devices, or Fog computing, requires the design of models that are energy-eﬃcient and compact [1–3].”
Model inversion attacks against collaborative inference (2019, methodology): “Collaborative inference can also provide better privacy protection, as the cloud now only receives the intermediate values instead of the raw data [43].”
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, background): “the edge and the cloud) and can be trained with multiple early exit points [161].”
Transformative effects of IoT, Blockchain and Artificial Intelligence on cloud computing: Evolution, vision, trends and open challenges (2019, background): “al [48] have demonstrated the use of edge and IoT end devices in realising a distributed training of a deep neural network.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “…[129] • Cooperative downlink transmission for communication-efﬁcient inference [130], [131] Device-edge joint inference: • Early exit: [132], [133] • Encoded transmission and pruning for compressing the transmitted data [134], [135] • Coded computing for cooperative edge inference [136]…”
Resource Management in Fog/Edge Computing (2018, background): “he edge while the EAB client merely displays the output. Neural Network-based: Recent research highlights the distribution of deep neural networks across user devices, edge nodes, and the cloud [66], [67]. The obvious beneﬁt is that the latency of inferring from a deep neural network is reduced for ”
DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters (2018, neutral): “In [9] and [10], DNN inference is partially offloaded to cloud servers to minimize processing latency and edge device resource consumption.”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “DDNNs [88] N/A Cloud-Edge-Device Level-1 • Latency • Accuracy • Model selection N/A • Latency reduction: over 20×”
Task Allocation for Mobile Federated and Offloaded Learning with Energy and Delay Constraints (2020, background): “[ 3], [16].”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, background): “coming data and the model in a shared-nothing setup. Simultaneously, various works [50, 51, 85] have examined the case where the client device can offload to other devices in the local network. Last, [73] also employs cloud-device synergy and progressive inference, but with a very different focus, i”
CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices (2020, methodology): “DDNN [44] leverages a similar principle and partitions DNN layers in a cloud-edge-device hierarchy.”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “[141] Image classification* Multi-camera multi-object detection [119] Distributed DNNs A, D Link”
JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services (2018, background): “the end-user device, edge, and cloud have been suggested [16] which are specialized for DNN applications.”
All One Needs to Know about Fog Computing and Related Edge Computing Paradigms: A Complete Survey (2018, neutral): “Harvard University researchers recently proposed a framework and its software implementation18 for distributed deep neural networks (DDNN), that can span over cloud, fog, and end devices [344].”