Editing Conditional Radiance Fields

By Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan C. Russell. IEEE International Conference on Computer Vision, 2021

300 citations recorded by the source corpus.

A neural radiance field (NeRF) is a scene model supporting high-quality view synthesis, optimized per scene. In this paper, we explore enabling user editing of a category-level NeRF – also known as a conditional radiance field – trained on a shape category. Specifically, we introduce a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region. First, we propose a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Observing multiple instances of the same category, our model learns underlying part semantics without any supervision, thereby allowing the propagation of coarse 2D user scribbles to the entire 3D region (e.g., chair seat). Next, we propose a hybrid network update strategy that targets specific network components, which balances efficiency and accuracy. During user interaction, we formulate an optimization problem that both satisfies the user’s constraints and preserves the original object structure. We demonstrate our editing approach on rendered views of three shape datasets and show that it outperforms prior neural editing approaches. Finally, we edit the appearance and shape of a single-view real photograph and show that the edit propagates to extrapolated novel views.

View Paper: https://www.semanticscholar.org/paper/61aa9c31b7d00c2bcd45b624f93a7952af20710c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “Ed-itNeRF [20] explores editing NeRFs by manipulating latent codes learned from object categories in a synthetic dataset.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, methodology): “As mentioned before, EditingNeRF [38] proposes to edit on the rendered image and uses network optimization to achieve editing transfer to the entire image and novel view images.”
TensoRF: Tensorial Radiance Fields (2022, background): “This representation has been quickly extended and applied in diverse graphics and vision applications, including generative models [6,26], appearance acquisition [2,4], surface reconstruction [40,27], fast rendering [33,47,12], appearance editing [42,19], dynamic capture [17,28].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “For existing NeRF-based scene editing / manipulation works [11, 22, 23, 57], Clip-NeRF [48], Object-NeRF [50], and LaTeRF [30] design approaches to change and complete objects represented by NeRFs; however, their application scenario is limited to simple objects, rather than scenes with significant…”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, methodology): “• Our feedforward code mappers enable the fast inference for editing different objects in the same category compared to the optimization-based editing method [21].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “They also used a deformation network similar to deformable NeRFs [40] (called instance-speciﬁc-network ) in Edit-NeRF [67] to help with modifying the scene based on latent space displacement.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Ed-itNeRF [24] stands as a pioneering work in this domain, as they edit both the shape and color of neural ﬁelds by conditioning them on latent codes.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Conditional [49, 37, 21, 63] and generative models [60, 61, 31] enable a degree”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Neural rendering with implicit representations has become a widely-used technique for solving many vision and graphics tasks ranging from view synthesis [5, 15, 25], to re-lighting [12,13], to pose and shape estimation [17,21,31], to 3D-aware image synthesis and editing [3, 11, 23], to modeling dyna”