CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

By Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa. ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, 2022

362 citations recorded by the source corpus.

We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8941e477b2f39eb92712f04400412da60d349ec1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Another branch of work employs CLIP to guide the text-conditioned generation of 3D objects [32, 41, 65, 83] or stylized meshes [45, 47] by encoding descriptive textual input.”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “Earlier works employed CLIP [63] guidance for 3D generation [31, 29, 56, 94] and manipulation [60, 40, 21] from text prompts.”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “CLIP-guided, optimization-based 3D generative models share a similar philosophy of optimizing 3D assets by guiding on 2D renderings [15, 17, 18, 22, 26, 33].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “…remains challenging due to the lack of large-scale aligned datasets of text and 3D. Alternative approaches use 2D vision-language models like CLIP [57] to create 3D content by formulating the generation as an optimization problem in the image domain [86, 29, 38, 48, 30] or as object alignment [69].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Furthermore, we compare our appearance modeling stage with several mesh stylization methods, namely Text2mesh [24], CLIP-Mesh [14] and Latent-NeRF [23].”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Building on this principle, Khalid et al. (2022) optimizes a mesh using a CLIP-guided objective, ﬁnding that the mesh representation is more efﬁcient to optimize than a NeRF.”
Text-to-3D using Gaussian Splatting (2023, methodology): “Early efforts in text-to-3D generation, including CLIP-forge [59], Dream Fields [29], Text2Mesh [42], TANGO [14], CLIPNeRF [72], and CLIP-Mesh [32], harness CLIP [53] guidance to create 3D assets.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “There are also models [18, 22, 29, 44, 46, 59, 75, 81] which adopt CLIP [56] to align each view of the 3D representation model with the text.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “With the recent explosion of generative image models [47, 40, 48, 17, 52, 53, 70, 15], there has been increasing interest in training similar generative models for other modalities such as audio [44, 10, 5, 31, 2, 25], video [24, 57, 23], and 3D assets [26, 28, 45, 41].”
TEXTure: Text-Guided Texturing of 3D Shapes (2023, methodology): “CLIP-Mesh de-forms an initial sphere with UV texture parameterization.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, methodology): “DreamField [16] and CLIPmesh [17] mitigate the training data issue by relying on a pre-trained image-text model [32] to optimize the underlying 3D representations (NeRFs and meshes), such that all 2D renderings reach high text-image alignment scores.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “To achieve open-vocabulary 3D generation, several methods propose to lift 2D image models for 3D generation (Jain et al., 2022; Poole et al., 2022; Wang et al., 2023a; Mohammad Khalid et al., 2022; Michel et al., 2022).”
MotionGPT: Human Motion as a Foreign Language (2023, neutral): “Recent years have witnessed a significant breakthrough in pre-trained large language models such as GPT [34, 35, 3, 27], BERT [7], and T5 [36, 5], which lead to the convergence of language [59, 47], image [33, 50, 20], mesh [55, 26] and mutlimodal [8] modeling.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Some works (Jain et al., 2022; Mohammad Khalid et al., 2022) optimize meshes or NeRFs to minimize the CLIP Radford et al. (2021) score between the rendered images and input prompt utilizing pretrained CLIP model.”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, methodology): “Also related to our work, CLIP-Mesh [15] and Dream Fields [12] use a pre-trained image-text model [28] to condition 3D generation on textual prompts and do not require 3D supervision.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Subsequently, some works [9, 13, 24] explore zero-shot text-guided 3D content creation utilizing the guidance from CLIP [33].”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “This challenge has prompted the development of text-to-3D methods [19, 31, 34, 20, 4, 29, 55, 16].”