CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

By Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao. Computer Vision and Pattern Recognition, 2021

478 citations recorded by the source corpus.

We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manip-ulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows individual control over both shape and appearance. This is achieved by performing the shape conditioning via applying a learned deformation field to the positional encoding and deferring color conditioning to the volumetric rendering stage. To bridge this disentangled latent representation to the CLIP embedding, we design two code mappers that take a CLIP embedding as input and update the latent codes to reflect the targeted editing. The mappers are trained with a CLIP-based matching loss to ensure the manipulation accuracy. Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images. We evaluate our approach by extensive experiments on a variety of text prompts and exemplar images and also provide an intuitive interface for interactive editing.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0483be6c3ec6cd41ffe248f86effc7468d3ac7be

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “…[58], Segment Anything [59], FastSAM [60], MobileSAM [61], Track Anything Model (TAM) [62] Open-Vocabulary 3D Scene Representation e.g. CLIP-NERF [63], LERF [64], DFF [65] Affordances e.g. Affordance Diffusion [66], VRB [67] Embodied AI e.g. Huang et al. [68], Statler [69], EmbodiedGPT [70],…”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Alternative approaches use 2D vision-language models like CLIP [57] to create 3D content by formulating the generation as an optimization problem in the image domain [86, 29, 38, 48, 30] or as object alignment [69].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “In the 3D domain, CLIP’s latent space provides a useful objective that enables semantic manipulation [40, 30, 45] where the domain gap is closed by a neural rendering.”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “These approaches allow for localized CLIP-guided edits, 3D spatial transformations [45], or localized scene removal [43] speciﬁed either by language or a reference image.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Although we can transform the scene in input or output space or via optimization-based editing [33, 84], this does not enable selective object-centric or semantic , local edits, such as moving a single object.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “There are also models [18, 22, 29, 44, 46, 59, 75, 81] which adopt CLIP [56] to align each view of the 3D representation model with the text.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “For existing NeRF-based scene editing / manipulation works [11, 22, 23, 57], Clip-NeRF [48], Object-NeRF [50], and LaTeRF [30] design approaches to change and complete objects represented by NeRFs; however, their application scenario is limited to simple objects, rather than scenes with significant…”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Additionally, some works [1, 10, 46, 47] leverage CLIP models to facilitate editing through the use of text prompts or reference images.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “3.3)”