Shap-E: Generating Conditional 3D Implicit Functions

By Heewoo Jun, Alex Nichol. arXiv.org, 2023

491 citations recorded by the source corpus.

We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at https://github.com/openai/shap-e.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dc7b2a39421f93ae134fbb43ef62763056ebe1c7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “Point-E [46] and Shap-E [30] tend to produce incompleted meshes.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “Imagine a scenario where one can inter-actively create and edit 3D content through simple verbal commands [22,30], bypassing the need for specialized software, or can instruct a robot to manipulate objects using nat-ural language [10].”
MVDream: Multi-view Diffusion for 3D Generation (2023, background): “It is pertinent to note ongoing research endeavors to reconstruct object shapes directly from monocular image inputs (Wu et al., 2023; Nichol et al., 2022; Jun & Nichol, 2023), aligning with the increasing stability of image generation techniques.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “…Time ↓ One-2-3-45 (Liu et al., 2023a) Inference-only 0.594 45 seconds Point-E (Nichol et al., 2022) Inference-only 0.587 78 seconds Shap-E (Jun & Nichol, 2023) Inference-only 0.591 27 seconds Zero-1-to-3 (Liu et al., 2023b) Optimization-based Table 1: Quantitative Comparisons on…”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, background): “Point-E [49] and Shape-E [50] also train for text-to-3D with the help of 3D models from an undisclosed source.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, methodology): “Some works attempt to train text-conditioned diffusion models on 3D representations such as point clouds and volumes [1,5,9,16,26,33,35,53,58,61].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “Figure 6: We compare One-2-3-45 with Point-E [52], Shap-E [29], Zero123 (Stable Dreamfusion version) [36], 3DFuse [68], and RealFusion [43].”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, background): “ControlNet [44], Stable Diffusion [46], InstructPix2Pix [47], and Shape-E [48]), and image understanding models (e.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86, 87].”
Structured 3D Latents for Scalable and Versatile 3D Generation (2024, unknown): “IV.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “It also has a better locality with respect to the image input compared to tokenizing the NeRF’s model weights as in Shap-E (Jun & Nichol, 2023).”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “With the rise of diffusion models, pioneer works have investigated image-conditioned 3D generative modeling on various representations, e.g ., point clouds [28, 32, 47, 55, 62], meshes [1, 25], SDF grids [6, 7, 44, 60] and neural fields [11, 16, 27, 31, 59].”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, unknown): “We have presented CLAY, a large-scale 3D generative model that supports multi-modal controls for high quality 3D asset generation, further bridging the gap between the vivid realms of human imagination and the tangible world of digital creation.”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “We compare Magic123 to recent methods (Point-E [58], ShapeE [34], 3DFuse [77], RealFusion [51], and Zero-1-to-3 [46]) for generating 3D objects from a single unposed image (the leftmost column).”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “The baselines include SOTA single-image 3D generation methods: ShapE [36], One-2-3-45 [54], One-2-3-45++ [53], DreamGaus-sian [93], Wonder3D [58], TriplaneGaussian [114], and LGM [92].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “3D content generation [120, 158, 210, 301] has been in high demand for a wide range of applications, including gaming, entertainment, and robotics simulation.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “…et al., 2022; Liu et al., 2023c; Zhou et al., 2021) or project the 3D models or multi-view rendered images into latent representations (Ntavelis et al., 2023; Zeng et al., 2022; Gupta et al., 2023; Jun & Nichol, 2023; Chen et al., 2023b) and perform the diffusion process in the latent space.”
A Survey on Generative Diffusion Models (2022, methodology): “In contrast, Shape-E [145] utilizes diffusion models for the NeRF representation of 3D objects.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Concurrently, there’s growing interest in reconstructing object shapes from monocular image inputs [14, 26, 48], demonstrating the evolving stability of image generation methodologies.”