NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

By Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, Hujun Bao. Computer Vision and Pattern Recognition, 2021

392 citations recorded by the source corpus.

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This de-sign allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both ac-curacy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time. Code is available at the project page: https://zju3dv.github.io/neuralrecon/.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8db0bf0fa406254d4cd57eb413443a0b96bd12b8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, methodology): “2 [47] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao.”
Neural RGB-D Surface Reconstruction (2021, methodology): “To reduce artifacts from classical reconstruction methods, a series of methods was proposed that use learned spatial priors to predict depth maps from color images [24, 28, 39], to learn multiview stereo using 3D CNNs on voxel grids [34, 68, 82], or multi-plane images [22], to reduce the influence o”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “A few recent papers [1, 4, 9, 27, 48, 57, 61] attempt to predict scene-level geometry with RGB-(D) inputs, but they all assume given camera poses.”
Continuous 3D Perception Model with Persistent State (2025, positive): “Our approach is related to learning based methods [14, 34, 74, 100], such as 3D-R2N2 [14], which utilize recurrent neural network architectures [21, 31, 33] for online 3D reconstruction.”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Different from depth estimation, 3D scene reconstruction methods [23, 37, 49, 6, 8] directly reconstruct a comprehensive and accurate 3D geometry of a scene.”