GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

By Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, Gordon Wetzstein. European Conference on Computer Vision, 2024

333 citations recorded by the source corpus.

We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0.1s. GRM is a feed-forward transformer-based model that efficiently incorporates multi-view information to translate the input pixels into pixel-aligned Gaussians, which are unprojected to create a set of densely distributed 3D Gaussians representing a scene. Together, our transformer architecture and the use of 3D Gaussians unlock a scalable and efficient reconstruction framework. Extensive experimental results demonstrate the superiority of our method over alternatives regarding both reconstruction quality and efficiency. We also showcase the potential of GRM in generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with existing multi-view diffusion models. Our project website is at: https://justimyhxu.github.io/projects/grm/.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e6cb30effd90d563bc680430ecc1317842d33b6e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Xintao Wang

In Computer Graphics and Computer-Aided Design

1. Weihao Cheng
2. Ying Shan
3. Jiale Xu
4. Yiming Gao
5. Shenghua Gao
6. Xintao Wang

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Xintao Wang
3. Weihao Cheng
4. Jiale Xu
5. Yiming Gao
6. Shenghua Gao
7. Chen Change Loy
8. Chao Dong

Citations in the corpus

• Recent advances in 3D Gaussian splatting (2024, unknown): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “Inspired by Instant3D, LGM [45] and GRM [58] replace the triplane NeRF [30] representation with 3D Gaussians [18] to enjoy its superior rendering efficiency and circumvent the need for memory-intensive volume rendering process.”
• A Survey on 3D Gaussian Splatting (2024, unknown): “The static version of 3D GS [111] fails to reconstruct dynamic scenes, resulting in a sharp drop in performance.”
• VGGT: Visual Geometry Grounded Transformer (2025, unknown): “Feed-forward Novel View Synthesis is progressing rapidly [4, 31, 34, 36, 70, 84, 95, 104].”