In-Place Scene Labelling and Understanding with Implicit Scene Representation

By Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, Andrew J. Davison. IEEE International Conference on Computer Vision, 2021

560 citations recorded by the source corpus.

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties.We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/be05b97c1a221b12640bac66cf4b322e3a81ca86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Urban Radiance Fields (2021, methodology): “Some research has been directed towards us-ing NeRF-like models for 3D reconstruction and novel view synthesis from images captured in natural environments (“in situ”) [19, 35, 60, 69, 75].”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “For example, Semantic NeRF [49] jointly encoded semantics with appearance and geometry within a NeRF for novel semantic view synthesis.”
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, unknown): “[30].”
Advances in Neural Rendering (2021, result): “While some prior work addresses this goal [KSW20,ZLLD21], this remains an open challenge.”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “In particular, SemanticNeRF [66] adds an extra head to NeRF to predict semantic category logits for any 3D position along with the usual density and color.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “…variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation [1,23,33,52,57,77,91].”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, background): “Various extensions have improved NeRF’s robustness to varying lighting conditions [37] or added supervision with depth [25, 51, 52], time-of-flight data [1], or semantic segmentation labels [56].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Semantic-NeRF [60] initiated the incorporation of semantic information into NeRFs and facilitated the generation of semantic masks from novel views.”
LERF: Language Embedded Radiance Fields (2023, methodology): “…prior works leverage this to improve the quality of 2D semantics, segmentations, or feature vectors by distilling them into 3D. Semantic NeRF [39] and Panoptic Lifting [33] embed semantic information from semantic segmentation networks into 3D, showing that combining noisy or sparse labels…”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, methodology): “We further note that conditioning via concatenation is a special case of a hypernetwork, namely a single linear layer outputting the biases of the ﬁrst LFN layer - see for instance appendix of [8] for a proof.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “[112] show that neural implicit representations can be combined with the supervision of semantic labels.”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, methodology): “3D Instance Segmentation We compare 3 methods for 3D instance segmentation on ScanNet++: PointGroup [16], HAIS [7] and SoftGroup [44] on 75 instances classes - semantic classes excluding wall, ceiling and floor.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “GS models, which update a large number of Gaussian points each training step, lack the memorization and moderating ability of neural networks.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, background): “Labelling Semantic-NeRF [70], NeSF [132], Fig-NeRF [68]”