MVDream: Multi-view Diffusion for 3D Generation

By Yichun Shi, Peng Wang, Jianglong Ye, M. Long, Kejie Li, X. Yang. International Conference on Learning Representations, 2023

1,054 citations recorded by the source corpus.

We introduce MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9aa01997226b5c4d705ae2e2f52c32681006654b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “As illustrated in Fig.1, ImageDream excels in generating objects with correct geometry from a given image, enabling users to leverage well-developed image generation models[29] for better image-text alignment than purely text-conditioned models like MVDream [37].”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, methodology): “Inspired by existing multi-view based models [32, 38], we also flatten the features before applying self-attention, allowing information to propagate among different cost volumes.”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, neutral): “Most works force the multi-view generated latents to communicate with each other by manipulating the self-attention layers with 3D-aware masks [36, 48, 88, 60, 95, 79, 78, 55].”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “…our sparse-view reconstruction model accepts free-viewpoint images as input, so we can integrate arbitrary multi-view generation model into our framework, e.g ., MVDream [43], ImageDream [51], SyncDreamer [24], SPAD [17] and SV3D [48], to achieve both text-to-3D and image-to-3D assets creation.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “MVDream [45], Consistent123 [53], and Wonder3D [27] also train multi-view diffusion models, yet still require post-processing for video rendering.”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “We choose Shap-E [36], Instant3D [46], LGM [92], and MVDream [81] as baselines.”
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model (2023, result): “11, we compare our results to MVDream [20] and Zero-1-to-3 XL [3, 12].”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, background): “We do so in a manner similar to [34], but only at the lowest UNet resolution, which maintains the computational cost very low.”
Recent advances in 3D Gaussian splatting (2024, neutral): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “To address this, more recent works such as Zero-123 [57], MVDream [81], and SyncDreamer [58] propose techniques to adapt and finetune pretrained image generation models such as Stable Diffusion (SD) for multi-view generation, thereby leveraging image priors from SD.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Fortunately, it’s possible to address these problems with recent advances in multi-view 2D diffusion models (Shi et al., 2023; Liu et al., 2023c; Zhao et al., 2023) and latent BRDF auto-encoder (Xu et al.”
Text-to-3D using Gaussian Splatting (2023, methodology): “For more qualitative comparisons and the performance of G SGEN under more advanced guidance including MVDream [63] and DeepFloyd IF [1], please refer to the appendix.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “Some other works improve the multi-view consistency across different novel views [37, 59], which directly enables NeRF optimization with pixel-wise loss with distillation, achieving impressive performance in few minutes.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “Here come two main streams for achieving this goal: (i) training a new diffusion model with 3D data [12, 15, 24, 48] (namely the 3D diffusion model ) and (ii) lifting the 2D diffusion model to 3D [2, 6, 9, 32, 52, 58, 67, 70, 76, 77, 81, 90].”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “In con-strast to GAN [4], some new works show that diffusion models can be employed in some 3D tasks, such as optical flow estimation [10,54], view synthesis [31,53,58], depth estimation [21,24,81], and normal estimation [32,35,45].”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, methodology): “Recently, several methods [20, 27, 43, 44, 51] propose to fine-tune multi-view diffusion models on 3D datasets to incorporate camera poses as an additional input.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, methodology): “Moreover, while our method delivers prompt results, MVDream [61] requires about 2 hours to generate a single shape.”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, methodology): “This stimulates the rapid developments of 3D content generation approaches [5, 14, 16, 21–24, 28, 30, 33, 34, 40, 45].”