LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching

By Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, Yingcong Chen. Computer Vision and Pattern Recognition, 2023

337 citations recorded by the source corpus.

The recent advancements in text-to-3D generation mark a significant milestone in generative models, unlocking new possibilities for creating imaginative 3D assets across var-ious real-world scenarios. While recent advancements in text-to-3D generation have shown promise, they often fall short in rendering detailed and high-quality 3D models. This problem is especially prevalent as many methods base themselves on Score Distillation Sampling (SDS). This paper identifies a notable deficiency in SDS, that it brings inconsistent and low-quality updating direction for the 3D model, causing the over-smoothing effect. To address this, we propose a novel approach called Interval Score Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes interval-based score matching to counteract over-smoothing. Furthermore, we incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline. Extensive experiments show that our model largely outperforms the state-of-the-art in quality and training efficiency. Our code is available at: EnVision-Research/LucidDreamer

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/6f709278506813d04a074e6fa20188cce9bb927b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yinghao Xu
2. Ceyuan Yang
3. Yujun Shen
4. Sida Peng
5. Hansheng Chen
6. Zifan Shi
7. Wang Yifan
8. Gordon Wetzstein

In Control and Systems Engineering

1. Sida Peng

Citations in the corpus

• Recent advances in 3D Gaussian splatting (2024, neutral): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
• A Survey on 3D Gaussian Splatting (2024, neutral): “Its ability to balance high-fidelity reconstruction with computational efficiency positions 3D GS as a cornerstone for advancing robotic perception, manipulation, and navigation in real-world environments.”
• GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”