4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

By Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, Xinggang Wang. Computer Vision and Pattern Recognition, 2023

1,495 citations recorded by the source corpus.

Representing and rendering dynamic scenes has been an important but challenging task. Especially, to accurately model complex motions, high efficiency is usually hard to guarantee. To achieve real-time dynamic scene rendering while also enjoying high training and storage efficiency, we propose 4D Gaussian Splatting (4D-GS) as a holistic representation for dynamic scenes rather than applying 3D-GS for each individual frame. In 4D-GS, a novel explicit representation containing both 3D Gaussians and 4D neural voxels is proposed. A decomposed neural voxel encoding algorithm inspired by HexPlane is proposed to efficiently build Gaussian features from 4D neural voxels and then a lightweight MLP is applied to predict Gaussian deformations at novel timestamps. Our 4D-GS method achieves real-time rendering under high resolutions, 82 FPS at an 800x800 resolution on an RTX 3090 GPU while maintaining comparable or better quality than previous state- of-the-art methods. More demos and code are available at https://guanjunwu.github.io/4dgs/.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/395bfdae59f1a5fde16213cade43d2587e4565df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting (2023, background): “Fortunately, recent work [11, 15, 44] with 3D Gaussian representation and tile-based splatting techniques has shown great superiority in the efficiency of high-resolution image rendering.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, positive): “Recently, 3D Gaussian splatting [15] has emerged as an alternative 3D representation to NeRF, showcasing impressive quality and speed in 3D and 4D reconstruction tasks [15, 26, 48, 51, 52].”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “To achieve real-time dynamic scene rendering, some studies [26, 40, 44] have extended the 3D Gaussian Splatting technique to dynamic scenes.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “While this method excels with dense input views and has achieved success in various 3D tasks [23, 39, 47], its reconstruction with sparse view inputs remains an open problem.”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, methodology): “We remark that the rotation of view directions is not considered in Wu et al. [45].”
Recent advances in 3D Gaussian splatting (2024, neutral): “Quantitative reconstruction results and sizes of 3D scenes after compression are shown in Table 2.”
A Survey on 3D Gaussian Splatting (2024, neutral): “Table 1) according to our taxonomy.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “For a fair comparison, we also migrate the graph-based method NSG [11] and dynamic Gaussians method 4DGS [17] to the nuScenes dataset.”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, methodology): “This approach has also been extended to model dynamic scenes [20, 44, 47, 48] with dense 6-DOF motion [20].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “It is also applied to dynamic scene reconstruction [28, 51, 55, 56] and 3D content generation [12, 42, 59].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, neutral): “…32.67 0.97 0.04 HexPlanes (Cao & Johnson, 2023) 31.04 0.97 0.04 K-Planes-explicit (Fridovich-Keil et al., 2023) 31.05 0.97 - K-Planes-hybrid (Fridovich-Keil et al., 2023) 31.61 0.97 - V4D (Gan et al., 2023) 33.72 0.98 0.02 4DGS (Wu et al., 2023) 1 Figure 3: Qualitative result on the flame salmon .”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Several follow-up methods [26, 52, 53] extend it to dynamic 3D scenes by tracking of all dense scene elements [26] or deformation field modeling [49, 52].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, neutral): “Concurrent work 4D-GS [50] is also compared since the official code has been released.”