Atlas: End-to-End 3D Scene Reconstruction from Posed Images

By Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, Andrew Rabinovich. European Conference on Computer Vision, 2020

349 citations recorded by the source corpus.

We present an end-to-end 3D reconstruction method for a scene by directly regressing a truncated signed distance function (TSDF) from a set of posed RGB images. Traditional approaches to 3D reconstruction rely on an intermediate representation of depth maps prior to estimating a full 3D model of a scene. We hypothesize that a direct regression to 3D is more effective. A 2D CNN extracts features from each image independently which are then back-projected and accumulated into a voxel volume using the camera intrinsics and extrinsics. After accumulation, a 3D CNN refines the accumulated features and predicts the TSDF values. Additionally, semantic segmentation of the 3D model is obtained without significant computation. This approach is evaluated on the Scannet dataset where we significantly outperform state-of-the-art baselines (deep multiview stereo followed by traditional TSDF fusion) both quantitatively and qualitatively. We compare our 3D semantic segmentation to prior methods that use a depth sensor since no previous work attempts the problem with only RGB input.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/35e2056e29d5295b2669fe696ae0c27007c94625

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “Following [30, 26], [53, 48, 14, 13] are fine-tuned on ScanNet.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “A few recent papers [1, 4, 9, 27, 48, 57, 61] attempt to predict scene-level geometry with RGB-(D) inputs, but they all assume given camera poses.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Most recently, top-performing methods returned back to explicit geometric reasoning [14, 27, 29, 32, 33, 34].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “We first conduct multi-camera 3D semantic occupancy prediction on nuScenes [7] dataset and compare with several state-of-the-art methods [8, 29, 22, 37].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, background): “Both PanopticFusion and ATLAS works only for static scenes, requires 3D supervision, and relies upon convolutions on a discrete voxel grid, which limits its resolution.”