Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices

By Olaf Kähler, Victor Adrian Prisacariu, Carl Yuheng Ren, Xin Sun, Philip H. S. Torr, David William Murray. 2015

322 citations recorded by the source corpus.

Volumetric methods provide efficient, flexible and simple ways of integrating multiple depth images into a full 3D model. They provide dense and photorealistic 3D reconstructions, and parallelised implementations on GPUs achieve real-time performance on modern graphics hardware. To run such methods on mobile devices, providing users with freedom of movement and instantaneous reconstruction feedback, remains challenging however. In this paper we present a range of modifications to existing volumetric integration methods based on voxel block hashing, considerably improving their performance and making them applicable to tablet computer applications. We present (i) optimisations for the basic data structure, and its allocation and integration; (ii) a highly optimised raycasting pipeline; and (iii) extensions to the camera tracker to incorporate IMU data. In total, our system thus achieves frame rates up 47 Hz on a Nvidia Shield Tablet and 910 Hz on a Nvidia GTX Titan XGPU, or even beyond 1.1 kHz without visualisation.

Subfields: Aerospace Engineering, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, neutral): “The RGB-D scanning system must be trivial to use, the data processing must be robust and automatic, the semantic annotations must be crowdsourced, and the ﬂow of data through the system must be handled by a tracking server.”
Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008–2017) (2018, background): “K¨ahler et al. [61] and Ondruska et al. [102] achieved dense and photo-realistic 3D reconstructions on tablet computers and mobile phones by highly optimizing the computational pipeline.”
Grasp Pose Detection in Point Clouds (2017, methodology): “In the active point cloud contingency, we obtained point clouds using the active sensing strategy where we run InﬁniTAM while moving the wrist-mounted sensor above the tray.”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, methodology): “In 3D, the representation of a scene is typically obtained from RGB-D surface reconstruction methods [4,5,6,7] which often store scanned geometry in a 3D voxel grid where the surface is encoded by an implicit surface function such as a signed distance field [8].”
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans (2018, background): “We now have 3D reconstruction frameworks, often based on volumetric fusion [6], that achieve impressive reconstruction quality [18, 29, 30, 41, 21] and reliable global pose alignment [41, 5, 8].”
Real-Time Large-Scale Dense 3D Reconstruction with Loop Closure (2016, methodology): “To integrate D t into the T-SDF we follow [17].”
A Robust 3D-2D Interactive Tool for Scene Segmentation and Annotation (2016, background): “HIGH-QUALITY 3D scene data has become increasingly available thanks to the growing popularity of consumer-grade depth sensors and tremendous progress in 3D scene reconstruction research [1], [2], [3], [4], [5], [6].”
Are We Ready for Service Robots? The OpenLORIS-Scene Datasets for Lifelong SLAM (2019, background): “InfiniTAM is a dense SLAM system based on point cloud matching with an iterative closed point (ICP) algorithm [20].”
Visual SLAM algorithms: a survey from 2010 to 2016 (2017, background): “achieve realtime processing of KinectFusion on mobile devices [58].”
Voxblox: Incremental 3D Euclidean Signed Distance Fields for on-board MAV planning (2016, result): “This is a common approach in other literature [10], [12], [17], [18].”
High precision grasp pose detection in dense clutter (2016, methodology): “One system was used to run our GPD algorithm, and we used InfiniTAM [6] on the other system to obtain a TSDF volume from the wristmounted sensor while moving the robot arm.”
ElasticFusion: Real-time dense SLAM and light source estimation (2016, background): “…per-surface-element-independent filtering by holding the camera pose fixed during the mapping step (Keller et al., 2013; Newcombe et al., 2011a), which has proven to be a highly efficient technique capable of running in real-time even on resource constrained mobile devices (Kahler et al., 2015).”
Structured Prediction of Unobserved Voxels from a Single Depth Image (2016, methodology): “It is worth noting that this ground truth dataset is only accurate up to the reconstruction error of [32].”
Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation (2022, background): “These poses are then coupled with their corresponding input point clouds to update a global map represented by geometric primitives such as cost volumes [20, 37], surfels [29, 35, 38] or voxels [12, 19, 21].”