Joint Semantic Segmentation and 3D Reconstruction from Monocular Video

By Abhijit Kundu, Yin Li, Frank Dellaert, Fuxin Li, James M. Rehg. 2014

254 citations recorded by the source corpus.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, background): “As no official pixel-wise annotations exist for KITTI, several independent groups have annotated approximately 700 frames [22, 29, 32, 33, 58, 64, 77, 80].”
3D Instance Segmentation of MVS Buildings (2021, methodology): “The existing multi-view semantic segmentation framework projects the 2D semantic labels back to the 3D model with the highest probability [44], [45].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, neutral): “depth or semantics [20, 29, 30].”
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016, neutral): “It should be noted that [147] and [219] focus only on the mapping part and they do not refine the early”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, background): “or in combination with 3D scene geometry [32, 17, 35].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “Typically, these approaches operate on a single RGB image; however, one can easily formulate the analogous task in 3D on a per-voxel basis [5, 13, 21, 34, 40, 41], which is a common scenario in the context of 3D scene reconstruction.”
Displets: Resolving stereo ambiguities using object knowledge (2015, background): “While the mutual benefits of recognition and reconstruction have been shown using simple priors and multiple views [15,29], little work has addressed the binocular stereo problem in this context with notable exceptions [33,48,61].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “Multi-view reconstruction and segmentation: Incorporating semantics into SLAM and SfM systems has a long history [2, 19, 27, 50].”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, methodology): “It has been used in robotics [1][2][3][4], medical applications [5][6], augmented reality [7], and most prominently automated driving [8][9][10][11].”
Multi-Label Semantic 3D Reconstruction Using Voxel Blocks (2016, methodology): “[14] solves a volumetric labeling problem using a conditional random field (CRF) formulation with image data from a single monocular camera which is mounted on a driving car.”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, methodology): “These tasks are traditionally handled individually [9], [12], or jointly handled offline with semantic SLAM [25] which could be time consuming.”
Dense Semantic 3D Reconstruction (2017, methodology): “The approach of [19] reconstructs street scenes from a moving monocular camera using a discrete graphbased formulation over a voxel grid.”
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation (2018, background): “This is especially problematic on the computationally heavy task of semantic segmentation [4,5,6,7,8,9,10].”
Tangent Convolutions for Dense Prediction in 3D (2018, background): “A popular line of work exploits graphical models, including conditional random fields [38, 11, 2, 58, 19, 27, 56].”
Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction (2015, methodology): “[11] proposed an offline method (based on personal communication) to integrate sparse (monocular) reconstruction with 2D semantic labels into a CRF model to determine the structure and labelling of a scene.”
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning (2018, methodology): “It has been used in robotics [60, 6, 62, 36], medical applications [14, 71], augmented reality [43], and most prominently automated driving [69, 53, 9, Table 4: Semantic Competition on Cityscapes [15].”
Meaningful maps with object-oriented semantic mapping (2016, methodology): “[16] derive a CRF model over 3D space, that jointly infers the semantic”
Large-Scale Semantic 3D Reconstruction: An Adaptive Multi-resolution Model for Multi-class Volumetric Labeling (2016, background): “Jointly inferring 3D geometry and semantics is a hard problem, and has only recently been tackled in a principled manner [17, 25, 36].”
Probabilistic data association for semantic SLAM (2017, methodology): “A global optimization for 3D reconstruction and semantic parsing has been proposed by [25], which is the closest work in semantic/geometric joint optmization.”