Edge Assisted Real-time Object Detection for Mobile Augmented Reality

By Luyang Liu, Hongyu Li, Marco Gruteser. ACM/IEEE International Conference on Mobile Computing and Networking, 2019

562 citations recorded by the source corpus.

Most existing Augmented Reality (AR) and Mixed Reality (MR) systems are able to understand the 3D geometry of the surroundings but lack the ability to detect and classify complex objects in the real world. Such capabilities can be enabled with deep Convolutional Neural Networks (CNN), but it remains difficult to execute large networks on mobile devices. Offloading object detection to the edge or cloud is also very challenging due to the stringent requirements on high detection accuracy and low end-to-end latency. The long latency of existing offloading techniques can significantly reduce the detection accuracy due to changes in the user's view. To address the problem, we design a system that enables high accuracy object detection for commodity AR/MR system running at 60fps. The system employs low latency offloading techniques, decouples the rendering pipeline from the offloading pipeline, and uses a fast object tracking method to maintain detection accuracy. The result shows that the system can improve the detection accuracy by 20.2%-34.8% for the object detection and human keypoint detection tasks, and only requires 2.24ms latency for object tracking on the AR device. Thus, the system leaves more time and computational resources to render virtual elements for the next frame and enables higher quality AR/MR experiences.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/63a9f550957049d25b35bff19e8c29d83da07821

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, background): “Interest (RoI) encoding [140] to focus on target objects in video frames can further reduce the bandwidth consumption and data transmission delay.”
A Full Dive Into Realizing the Edge-Enabled Metaverse: Visions, Enabling Technologies, and Challenges (2022, background): “However, resources on the edge networks are limited, while the customization and communication resources used for AR adaptation are enormous [97], [98], [99].”
Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach (2021, unknown): “However, the transmission latency remains a bottleneck for applications with stringent delay requirements [4], [15], [16], as a huge volume of data (e.g., 3D images, high-deﬁnition videos, and point cloud data) need to be uploaded.”
Deep Learning With Edge Computing: A Review (2019, methodology): “, as used in [33]), with Nvidia-provided SDKs used to program the devices.”
A Survey on Mobile Augmented Reality With 5G Mobile Edge Computing: Architectures, Applications, and Technical Aspects (2021, background): “Multi-access Edge Computing (MEC) brings the computations closer to the end devices, minimizing the latency of cloud ofﬂoading and facilitating the latency-critical applications [42]–[45].”