Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

By Zi-Xin Zou, Zhipeng Yu, Yuanchen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, Song-Hai Zhang. Computer Vision and Pattern Recognition, 2023

324 citations recorded by the source corpus.

Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation ofdiffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at https://zouzx.github.io/TriplaneGaussian/

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology

View Paper: https://www.semanticscholar.org/paper/536dcef9e5f22bb694bb154120050bfed07f8e90

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Yinghao Xu
2. Jiaxiang Tang
3. Gang Zeng
4. Gordon Wetzstein
5. Sida Peng
6. Zhaoxi Chen
7. Yujun Shen
8. Ziwei Liu

In Computer Graphics and Computer-Aided Design

1. Weihao Cheng
2. Yinghao Xu
3. Ying Shan
4. Jiaxiang Tang
5. Gang Zeng
6. Gordon Wetzstein
7. Sida Peng
8. Zhaoxi Chen

In Geology

1. Yinghao Xu
2. Gordon Wetzstein
3. Zhaoxi Chen
4. Ziwei Liu

Citations in the corpus

• Recent advances in 3D Gaussian splatting (2024, unknown): “The U-Net is asymmetric, which allows for the input of high-resolution images while limiting the number of output Gaussians.”
• LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, methodology): “For a collection of 30 images, we render 360-degree rotating videos of the 3D Gaussians generated from DreamGaussain [47] (only the first stage), TriplaneGaussian [62], and ours.”
• Structured 3D Latents for Scalable and Versatile 3D Generation (2024, unknown): “More Implementation Details”
• GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “Reconstruction methods, e.g., TriplaneGaussian [114] struggle to realistically complete the unseen region (rows 1–2).”
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “Different from view-space reconstruction models [13, 45, 46, 63], our model reconstruct 3D objects in a canonical world space where the z -axis aligns with the anti-gravity direction.”
• CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, unknown): “For example, Instant3D [Li et al. 2023], LRM [Hong et al. 2024; Wang et al. 2024], DMV3D [Xu et al. 2024] and TGS [Zou et al. 2024] further utilized sparse-view or single-view reconstructors that leverage a Vision Transformer (ViT) as the vision backbone, coupled with a deep transformer…”