MonoScene: Monocular 3D Semantic Scene Completion

By Anh-Quan Cao, Raoul de Charette. 2021

497 citations recorded by the source corpus.

MonoScene proposes a 3D Semantic Scene Completion (SSC) framework, where the dense geometry and semantics of a scene are inferred from a single monocular RGB image. Different from the SSC literature, relying on 2.5 or 3D input, we solve the complex problem of 2D to 3D scene reconstruction while jointly inferring its semantics. Our framework relies on successive 2D and 3D UNets, bridged by a novel 2D-3D features projection inspired by optics, and introduces a 3D context relation prior to enforce spatio-semantic consistency. Along with architectural contributions, we introduce novel global scene and local frustums losses. Experiments show we outperform the literature on all metries and datasets while hallucinating plausible scenery even beyond the camera field of view. Our code and trained models are available at https://github.com/cv-rits/MonoScene.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Justin Johnson
2. Zhengbiao Zhu
3. Georgia Gkioxari
4. Dalong Du
5. Yunpeng Zhang
6. Jiwen Lu
7. Jie Zhou
8. Jitendra Malik

In Computer Graphics and Computer-Aided Design

1. Justin Johnson
2. Georgia Gkioxari
3. Jiwen Lu
4. Jie Zhou
5. Jitendra Malik
6. Zhengbiao Zhu
7. Yunpeng Zhang
8. Christopher Bongsoo Choy

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Jitendra Malik
4. Xiaogang Wang
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Alan L. Yuille
8. Andrew Zisserman

In 3D Vision & Pose

1. Jitendra Malik
2. Georgia Gkioxari
3. Justin Johnson
4. Christopher Bongsoo Choy
5. Sanja Fidler
6. Christoph Feichtenhofer
7. Chaozheng Wu
8. Chen Feng

Citations in the corpus

• Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For scenes, techniques include plane fitting [38], 3D model fitting and retrieval [14, 39] or leverage symmetries [27] and predict 3D semantics [3, 13, 58].”
• SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “We first conduct multi-camera 3D semantic occupancy prediction on nuScenes [7] dataset and compare with several state-of-the-art methods [8, 29, 22, 37].”
• VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, methodology): “This motivated the study of camera-based SSC solutions, as first proposed in the pioneering work of MonoScene [4].”
• OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “OccFormer is evaluated by its task of semantic scene completion, but with the monocular left camera as input following MonoScene [4].”
• Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “We formalize the 3D occupancy prediction task as follows: a model needs to jointly estimate the occupancy state and semantic label of every voxel in the scene from images [2, 24, 5].”