Text-to-3D using Gaussian Splatting

By Zilong Chen, Feng Wang, Huaping Liu. Computer Vision and Pattern Recognition, 2023

384 citations recorded by the source corpus.

Automatic text-to-3D generation that combines Score Distillation Sampling (SDS) with the optimization of volume rendering has achieved remarkable progress in synthesizing realistic 3D objects. Yet most existing text-to-3D methods by SDS and volume rendering suffer from inaccurate geometry, e.g., the Janus issue, since it is hard to explicitly integrate 3D priors into implicit 3D representations. Besides, it is usually time-consuming for them to generate elaborate 3D models with rich colors. In response, this paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting that enables the incorporation of 3D prior. Specifically, our method adopts a progressive optimization strategy, which includes a geometry optimization stage and an appearance refinement stage. In geometry optimization, a coarse representation is established under 3D point cloud diffusion prior along with the ordinary 2D SDS optimization, ensuring a sensible and 3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an iterative appearance refinement to enrich texture details. In this stage, we increase the number of Gaussians by compactness-based densification to enhance continuity and improve fidelity. With these designs, our approach can generate 3D assets with delicate details and accurate geometry. Extensive evaluations demonstrate the effectiveness of our method, especially for capturing high-frequency components. Our code is available at https://github.com/gsgen3d/gsgen.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/86b5318b0a69ccdeec17abb0120e4bd7688a4b59

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on 3D Gaussian Splatting (2024, positive): “We also provide a collection of representative datasets ( c.f .”
LangSplat: 3D Language Gaussian Splatting (2023, background): “Mean-while, some researchers have combined 3D Gaussian Splat-ting with diffusion models to achieve efficient text-to-3D generation [10, 38, 46].”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, background): “The other some [2, 6, 40] focus on addressing the Janus problems, e.g., MVDream [40] propose to fine-tune the pre-trained diffusion models to make it 3D aware.”
GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting (2023, background): “This promising technology has been rapidly applied in several fields, including 3D generation [3, 33, 48], dynamic scene modeling [15][44][46], and photorealistic drivable avatars [52].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, positive): “Moreover, it has been observed in [7, 45, 53] that optimizing Gaussian Splatting (GS) using highly random generative guidance like Score Distillation Sampling [36] poses signiﬁcant challenges.”
Recent advances in 3D Gaussian splatting (2024, positive): “In addition to 2D SDS, GSGEN [158] introduces a 3D SDS loss based on Point-E [159], a text-to-point-cloud diffusion model, to mitigate the multi-face or Janus problem.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “GEN [9] implements high-quality generation from text to 3D.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “It is also applied to dynamic scene reconstruction [28, 51, 55, 56] and 3D content generation [12, 42, 59].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Another stream of works [6, 43, 54] focus on the 3D content creation, which combine Gaussian Splatting with the diffusion-based models and show high-quality generation results.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, methodology): “GSGen [10] and GaussianDreamer [59] explore various densification and initialization strategies for text to 3D Gaussians generation.”