Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

By Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, Zhaopeng Cui. IEEE International Conference on Computer Vision, 2021

309 citations recorded by the source corpus.

Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene branch encodes the scene geometry and appearance, and the object branch encodes each standalone object conditioned on learnable object activation codes. To survive the training in heavily cluttered scenes, we propose a scene-guided training strategy to solve the 3D space ambiguity in the occluded regions and learn sharp boundaries for each object. Extensive experiments demonstrate that our system not only achieves competitive performance for static scene novel-view synthesis, but also produces realistic rendering for object-level editing.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8bdfd48579130b50112c03170904d803aafe6ba9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “Previous object-aware frameworks [40, 61] used a shared MLP with instance-specific latent codes to incorporate this prior.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, unknown): “Efforts have been made to adapt NeRF to dynamic scenes [29,30,15], and to even edit and compose scenes with various NeRF models [50,9], widening their potential application.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, background): “These issues could be addressed by learning transient objects [38] or directly modeling dynamic objects [43, 68].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “For existing NeRF-based scene editing / manipulation works [11, 22, 23, 57], Clip-NeRF [48], Object-NeRF [50], and LaTeRF [30] design approaches to change and complete objects represented by NeRFs; however, their application scenario is limited to simple objects, rather than scenes with significant…”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “3.5) Foreground/Background NeRF-W (2020) [66], NeRF++ (2020) [68], GIRAFFE (2020) [61] Semantic/ Object Composition Fig-NeRF (2021) [69], Yang et al. (2021) [70], Semantic-NeRF (2021) [71], Panoptic Neural Fields (2022) [72] Pose Estimation (Sec.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, methodology): “A recent work, ObjectNeRF [74], proposed to learn a decompositional neural radiance field, which separates the objects and the background.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “[104] used ground-truth instance segmentation masks and trained multiple networks, although the authors did not investigate this issue.”
Urban Radiance Fields (2021, methodology): “Some research has been directed towards us-ing NeRF-like models for 3D reconstruction and novel view synthesis from images captured in natural environments (“in situ”) [19, 35, 60, 69, 75].”