SyncDreamer: Generating Multiview-consistent Images from a Single-view Image

By Yuan Liu, Chu-Hsing Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, Wenping Wang. International Conference on Learning Representations, 2023

747 citations recorded by the source corpus.

In this paper, we present a novel diffusion model called that generates multiview-consistent images from a single-view image. Using pretrained large-scale 2D diffusion models, recent work Zero123 demonstrates the ability to generate plausible novel views from a single-view image of an object. However, maintaining consistency in geometry and colors for the generated images remains a challenge. To address this issue, we propose a synchronized multiview diffusion model that models the joint probability distribution of multiview images, enabling the generation of multiview-consistent images in a single reverse process. SyncDreamer synchronizes the intermediate states of all the generated images at every step of the reverse process through a 3D-aware feature attention mechanism that correlates the corresponding features across different views. Experiments show that SyncDreamer generates images with high consistency across different views, thus making it well-suited for various 3D generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fcd0de4066d93fa3822a14898008fa2dd99f7be6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model (2023, methodology): “Recent works like One-2-3-45 [11], SyncDreamer [13] and Consis-tent123 [10] build extra layers upon Zero-1-to-3 to obtain more 3D-consistent results.”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, positive): “These features are back-projected into a 3D volume similar to previous method [Liu et al. 2024a], then down-sampled and flattened for integration into the DiT using cross-attention, a similar procedure to the voxel condition.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, neutral): “Some single-image-to-3D meth-ods [13, 19, 30, 33–36, 38, 54, 60, 65, 66, 71, 78, 79, 82, 83] also employ similar ideas.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “A dataset of 30 images collected from previous works (Melas-Kyriazi et al., 2023; Liu et al., 2023a; Tang et al., 2023b; Liu et al., 2023c) and Internet covering various objects is used.”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “…our sparse-view reconstruction model accepts free-viewpoint images as input, so we can integrate arbitrary multi-view generation model into our framework, e.g ., MVDream [43], ImageDream [51], SyncDreamer [24], SPAD [17] and SV3D [48], to achieve both text-to-3D and image-to-3D assets creation.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “Methods like MVDream [43], ImageDream [9], Zero123++ [44], ConsistNet [45], SyncDreamer [46] and ViewDiff [47] fine-tune text-to-image models to generate multiple views simultaneously.”
Recent advances in 3D Gaussian splatting (2024, neutral): “The U-Net is asymmetric, which allows for the input of high-resolution images while limiting the number of output Gaussians.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “To address the inconsistency issue in prior works, MVDream [45], SyncDreamer [26], HexGen3D [29], and Zero123++ [44] propose to generate multiple (specific) views of an object simultaneously.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “Some other works improve the multi-view consistency across different novel views [37, 59], which directly enables NeRF optimization with pixel-wise loss with distillation, achieving impressive performance in few minutes.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, methodology): “Within the optimization-based approaches, our baselines include DreamFusion [49] with Zero123 XL [34] as its backbone, as well as SyncDreamer [36], and DreamGaussian [63].”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “SyncDreamer[36] generates multi-view consistent images from a single image for deriving 3D geometry.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “In addition, the temporal attention layers in our video model naturally assist in the generation of consistent multi-views of an object without needing any explicit 3D structures like in [58].”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, background): “Synthesizing 3D consistent multi-view images using 2D diffusion models has been explored by many works [25,43,44,51].”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, background): “More recent works use diffusion models, following their immense success on image generation [5, 15, 16, 24, 25, 29, 30, 36, 40, 48, 53, 54, 56, 66].”