CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

By Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero. Computer Vision and Pattern Recognition, 2021

362 citations recorded by the source corpus.

Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape gener-ation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive compar-ative evaluations to better understand its behavior.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/738e3e0623054da29dc57fc6aee5e6711867c4e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “In addition to optimization-based 3D shape generation, some works train a 3D generative model but leverage the embedding space of CLIP [8, 39, 67], and some works focus on generating textures or materials for input meshes using 2D models’ prior [47, 76, 7, 46, 63].”
Multimodal Learning With Transformers: A Survey (2022, background): “graph [228], video to caption [229], [230], [231], [232], image to caption [233], [234], [235], [236], [237], text to speech [238], text to image [205], [239], text to shape [240], RGB to 3D human”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, neutral): “Early works [98, 76, 106, 110] based on Variational Auto-encoder [41], generative adversarial networks (GANs) [29], normalizing flow [65], and auto-regressive modeling [31] demonstrated strong generative capabilities within several specific categories.”
Text2Mesh: Text-Driven Neural Stylization for Meshes (2021, methodology): “Another concurrent work uses the ShapeNet dataset [5] and CLIP to perform unconditional 3D voxel generation [48].”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “Another branch of work employs CLIP to guide the text-conditioned generation of 3D objects [32, 41, 65, 83] or stylized meshes [45, 47] by encoding descriptive textual input.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…learn directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86,…”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, methodology): “F. 11 we condition LION on CLIP embeddings of the shapes’ rendered images, following CLIP-Forge [34] (Fig.”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “More recently, CLIP [40] was shown as a powerful conditioner for generative 3D models [43,30].”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “CLIP-Forge [49] generates multiple object geometries from text prompts using a CLIP embeddingconditioned normalizing flow model and geometry-only decoder trained on ShapeNet categories.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “CLIP also provides powerful means for supporting textdriven image manipulation with Generative Adversarial Networks (GANs) or other generative models [8, 54, 58].”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Inspired by these results, recent works have explored text-conditional generation in other modalities, such as video (Hong et al., 2022; Singer et al., 2022; Ho et al., 2022b;a) and 3D objects (Jain et al., 2021; Poole et al., 2022; Lin et al., 2022a; Sanghi et al., 2021; 2022).”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, methodology): “CLIP-Forge [41] consists of two separate components, an implicit autoencoder conditioned on shape codes, and a normalizing ﬂow model that is trained to generate shape codes according to CLIP embeddings.”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Alternative approaches use 2D vision-language models like CLIP [57] to create 3D content by formulating the generation as an optimization problem in the image domain [86, 29, 38, 48, 30] or as object alignment [69].”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “There are also models [18, 22, 29, 44, 46, 59, 75, 81] which adopt CLIP [56] to align each view of the 3D representation model with the text.”
DreamFusion: Text-to-3D using 2D Diffusion (2022, methodology): “CLIP has been used to guide other approaches based on voxel grids and meshes (Sanghi et al., 2022; Jetchev, 2021; Wang et al., 2022).”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “In the 3D domain, CLIP’s latent space provides a useful objective that enables semantic manipulation [40, 30, 45] where the domain gap is closed by a neural rendering.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “Sanghi et al. [54] encode voxel grids into latents which are used to condition an implicit occupancy network.”
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models (2022, methodology): “CLIP-Forge [Sanghi et al. 2021] alleviates the issue of paired text and 3D models by relying only on the 3D models to train an encoder and decoder and then guiding generation of the decoder with CLIP to produce results that match a text prompt, this only partially solves the problem because now the ”