Decomposing NeRF for Editing via Feature Field Distillation

By Sosuke Kobayashi, Eiichi Matsumoto, Vincent Sitzmann. Neural Information Processing Systems, 2022

461 citations recorded by the source corpus.

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6115a34ad9dd868fe886df815071023fd432e7a1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, methodology): “Methods such as Distilled Feature Fields [15] and Neural Feature Fusion Fields [43] distill 2D features from pre-trained models such as LSeg [17] and DINO [4] into the radiance ﬁelds, which enable speciﬁcation of regions.”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “Some early attempts to construct 3D feature fields included Distilled Feature Fields [20] and Neural Feature Fusion Fields [39].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Additionally, 3D editing techniques involve combining 2D image manipulation, such as inpainting, with neural ﬁelds training [19, 23].”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, methodology): “DFF [35], NeRF-SOS [18], INS [17], SA3D [9] distill various pre-trained 2D image feature ex-tractors to perform corresponding tasks in the 3D domain.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “Building upon it, subsequent research focusing on in-domain scene modeling, has expanded the scope by introducing instance modeling [9, 20, 39, 47] or encoding visual features [17, 35, 44–46] that support semantic delineation.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “According to the GLO authors, this method can be thought of as a Discriminator-less GAN.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, background): “A rich body of prior work in vision identifies that different types of visual representations trained with different inductive biases can lead to improved performance for a broad spectrum of applications (Kobayashi et al., 2022; Karamcheti et al., 2023).”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “Earlier works employed CLIP [63] guidance for 3D generation [31, 29, 56, 94] and manipulation [60, 40, 21] from text prompts.”
LERF: Language Embedded Radiance Fields (2023, methodology): “We compare against distilling LSeg features into 3D as in DFF [20], but implemented in our own codebase for an apples-to-apples comparison.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “Recent success of 2D open-vocabulary segmentation models such as LSeg [39], OpenSeg [18], and OV-Seg[41] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 10, 17, 21, 27, 29, 32, 34, 45, 49, 56, 57].”
gsplat: An Open-Source Library for Gaussian Splatting (2024, unknown): “This is motivated by algorithms that combine learned feature maps with diﬀerentiable volume rendering (Kobayashi et al., 2022; Kerr et al., 2023).”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, methodology): “The seminal work, Neural Radiance Fields (NeRF) [21], has been widely studied and advanced in a variety of applications [2, 3, 32, 19, 23, 13, 25], including novel view synthesis [21, 18], 3D generation [25, 10], deformation [23, 26, 28], video [15, 35, 7, 24, 14].”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “5, we contrast our method with NeRF-DFF [18].”