Text2Mesh: Text-Driven Neural Stylization for Meshes

By Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, Rana Hanocka. Computer Vision and Pattern Recognition, 2021

452 citations recorded by the source corpus.

In this work, we develop intuitive controls for editing the style of 3D objects. Our framework, Text2Mesh, stylizes a 3D mesh by predicting color and local geometric details which conform to a target text prompt. We consider a disentangled representation of a 3D object using a fixed mesh input (content) coupled with a learned neural network, which we term a neural style field network (NSF). In order to modify style, we obtain a similarity score between a text prompt (describing style) and a stylized mesh by harnessing the representational power of CLIP. Text2Mesh requires neither a pre-trained generative model nor a specialized 3D mesh dataset. It can handle low-quality meshes (non-manifold, boundaries, etc.) with arbitrary genus, and does not require UV parameterization. We demonstrate the ability of our technique to synthesize a myriad of styles over a wide variety of 3D meshes. Our code and results are available in our project webpage: https://threedle.github.io/text2meshl.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/d15b27edf3630728cdb40f49946365d9011641cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Control and Systems Engineering

Citations in the corpus

Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Related methods refine existing 3D input through text guidance in a similar fashion [46, 10, 87, 62].”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “The results and comparison against a SOTA text-driven mesh stylization method, Text2Mesh [44], are provided in Fig.”
Text-to-3D using Gaussian Splatting (2023, unknown): “, 2022), Text2Mesh (Michel et al., 2022), TANGO (Chen et al.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Another branch of work employs CLIP to guide the text-conditioned generation of 3D objects [32, 41, 65, 83] or stylized meshes [45, 47] by encoding descriptive textual input.”
DreamFusion: Text-to-3D using 2D Diffusion (2022, unknown): “As in previous work on text-to-3D generation (Hong et al., 2022; Michel et al., 2021), we find it beneficial to randomly replace the albedo color ρ with white (1, 1, 1) to produce a “textureless” shaded output.”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “Earlier attempts at 3D reconstruction leveraged 3D priors like topology constraints to assist in 3D generation [88, 53, 59, 24].”
TEXTure: Text-Guided Texturing of 3D Shapes (2023, methodology): “Recent works have begun making significant progress in painting and texturing 3D objects by using language-image models as guidance [9, 28, 30, 51].”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “, 2022) and even texture synthesis for 3D objects (Michel et al., 2021).”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “In addition to optimization-based 3D shape generation, some works train a 3D generative model but leverage the embedding space of CLIP [8, 39, 67], and some works focus on generating textures or materials for input meshes using 2D models’ prior [47, 76, 7, 46, 63].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “In the 3D domain, CLIP’s latent space provides a useful objective that enables semantic manipulation [Sanghi et al. 2021; Michel et al. 2021; Wang et al. 2021a] where the domain gap is closed by a neural rendering.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “To achieve open-vocabulary 3D generation, several methods propose to lift 2D image models for 3D generation (Jain et al., 2022; Poole et al., 2022; Wang et al., 2023a; Mohammad Khalid et al., 2022; Michel et al., 2022).”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “More recently, CLIP [43] was shown as a powerful conditioner for generative 3D models [47,33].”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, methodology): “We stress, however, that our approach is not specific to either model and that subsequent work has already shown success that builds on our work using other models [6, 13, 30, 50], and even in other modalities [19, 54].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Furthermore, we compare our appearance modeling stage with several mesh stylization methods, namely Text2mesh [24], CLIP-Mesh [14] and Latent-NeRF [23].”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “CLIP-guided, optimization-based 3D generative models share a similar philosophy of optimizing 3D assets by guiding on 2D renderings [14, 16, 17, 20, 25, 33].”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, methodology): “Albeit promising, SDS has shown over-smoothing effects in a lot of literatures [21, 30, 33, 49].”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “To increase usability (and as explored in other 3D domains such as meshes [24, 10]), ClipNeRF [45] and NeRF-Art [46] extend this line of work by encouraging similarity between CLIP embeddings of the scene and a short text prompt.”