NeRF-Editing: Geometry Editing of Neural Radiance Fields

By Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, Lin Gao. Computer Vision and Pattern Recognition, 2022

299 citations recorded by the source corpus.

Implicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes the novel view images of the edited scene without re-training the network. Specifically, we establish a correspondence between the extracted explicit mesh representation and the implicit neural representation of the target scene. Users can first utilize well-developed mesh-based deformation methods to deform the mesh representation of the scene. Our method then utilizes user edits from the mesh representation to bend the camera rays by introducing a tetrahedra mesh as a proxy, obtaining the rendering results of the edited scene. Extensive experiments demonstrate that our framework can achieve ideal editing results not only on synthetic data, but also on real scenes captured by users.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e735a4aad6979c937f5694a2c61231c9b99254d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “Alternatively, one can specify bounding boxes [30, 49], to allow easy compositing of different objects [51] as well as spatial manipulations and geometry deformations [50].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, methodology): “…field and query the transformation of each control point p k at each timestep t as: Note that in practical implementations, R ti is represented equivalently as a quaternion r ti for more stable optimization and convenient interpolation for generating the motions of Gaussians in the follow-up steps.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “For existing NeRF-based scene editing / manipulation works [11, 22, 23, 57], Clip-NeRF [48], Object-NeRF [50], and LaTeRF [30] design approaches to change and complete objects represented by NeRFs; however, their application scenario is limited to simple objects, rather than scenes with significant…”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Geometry-based methods [20, 49, 50, 55] translate neural ﬁelds into meshes and synchronize mesh deformation with implicit ﬁelds.”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, methodology): “Subsequent studies have adhered to the aforementioned design when exploring applications related to NeRF deformations, such as static scene editing and dynamic scene reconstruction [5, 19, 21, 28, 31, 32, 51].”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, background): “We also do not model animation ( e.g ., turn signals, traffic lights) or actor deformation ( e.g . walking) and hope to incorporate ideas from works [57,59,90,99] that tackle this.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “Constructing radiance fields for dynamic scenes is a critical branch in the advancement of NeRF, with significant implications for real-world applications.”