Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

By Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa. IEEE International Conference on Computer Vision, 2023

590 citations recorded by the source corpus.

We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work. Result videos can be found on the project website: https://instruct-nerf2nerf.github.io.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/26c22380282a00166273038bc5ba785d845d61ad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “3D object style transfer results comparison between Instruct-NeRF2NeRF [10] and our Gaussian Grouping under the same instruction prompt “Turn the bear into a panda” using InstructPix2Pix [2].”
Recent advances in 3D Gaussian splatting (2024, unknown): “…appearance side, GaussianEditor [92] proposes to first modify 2D images with language input with diffusion model [93] in the masked region generated by the recent 2D segmentation model [80] and updating attributes of Gaussians again similar to the previous NeRF editing work Instruct-NeRF2NeRF [94].”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, unknown): “Please refer to the supplementary materials for more implementation details.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Besides, Instruct-Nerf2Nerf [12] typically requires more than 30 minutes to complete the editing of a scene, whereas our method iN2N [12] only takes between 5 to 10 minutes.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Research in this space has aimed to improve distillation strategies [18, 19, 20, 21, 22], swap in other 3D representations [23, 24, 25, 26, 27], and amortize the optimization process [28].”