SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation

By Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Gerhard Schwing, Liangyan Gui. Computer Vision and Pattern Recognition, 2022

371 citations recorded by the source corpus.

In this work, we present a novel framework built to sim-plify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, in-cluding images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multimodal inputs, we employ task-specific encoders with dropout followed by a cross-attention mechanism. Due to its flexibility, our model naturally supports a variety of tasks, outperforming prior works on shape completion, image-based 3D reconstruction, and text-to-3D. Most interestingly, our model can combine all these tasks into one swiss-army-knife tool, enabling the user to perform shape generation using incomplete shapes, images, and textual descriptions at the same time, providing the relative weights for each input and facilitating interactivity. Despite our approach being shape-only, we further show an efficient method to texture the generated shape using large-scale text-to-image models.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/02c0b857b13030a596bd34dc0d75f499aaf4b420

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “Modern zero-shot text-to-image generators [66, 71, 65, 72, 2, 19] allowed to improve these results by providing stronger synthesis priors [62, 86, 52, 10, 12].”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “With the rise of diffusion models, pioneer works have investigated image-conditioned 3D generative modeling on various representations, e.g ., point clouds [28, 32, 47, 55, 62], meshes [1, 25], SDF grids [6, 7, 44, 60] and neural fields [11, 16, 27, 31, 59].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. els [8, 16, 37, 87], a majority rely solely on 3D shape datasets for training.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Many recent works [1, 7, 10, 17, 23, 27, 30, 31, 32, 34, 42, 45, 46, 48, 75, 87, 88] try to repeat the success of diffusion models on the 3D generation task.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “In addition to optimization-based 3D shape generation, some works train a 3D generative model but leverage the embedding space of CLIP [8, 39, 67], and some works focus on generating textures or materials for input meshes using 2D models’ prior [47, 76, 7, 46, 63].”
Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “More Implementation Details”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, background): “…diffusion models usually work on a single object category and suffer from limited diversity (Jun & Nichol, 2023; Nichol et al., 2022; Gupta et al., 2023; Lorraine et al., 2023; Zhang et al., 2023; Zheng et al., 2023; Ntavelis et al., 2023; Chen et al., 2023b; Cheng et al., 2023; Gao et al., 2022).”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, neutral): “Various implicit generative methods have shown impressive performance using adversarial [6, 53] and diffusion-based [8, 14, 41] models.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “They aim to exploit techniques such as GAN [16], diffusion model [8, 27, 41, 78, 81] to sample a high 3D content from pre-trained distribution.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, background): “There exists another group of works that endeavor to directly produce 3D geometries like point clouds [41, 45, 75, 80], meshes [16, 37], neural fields [1, 4, 8, 15, 17, 21, 26– 28, 44, 46, 65, 76] via network inference to avoid time-consuming per-shape optimization.”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “The success of diffusion models [33] and their variants [58, 53] in text-conditioned image [74, 45] generation has spurred the popularity of diffusion-based shape generative models, with notable works [16, 119] achieving stable cross-category shape generation.”
Text-to-3D using Gaussian Splatting (2023, background): “Another line of work lies in generating 3D assets directly through a 3D diffusion model based on NeRF or other differentiable representations (Wang et al., 2023b; Jun & Nichol, 2023; Liu et al., 2023a; Cheng et al., 2023).”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Other approaches train custom diffusion models on a similar textto-3D task [39, 49, 12].”