Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

By Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, Li Zhang. International Conference on Learning Representations, 2023

581 citations recorded by the source corpus.

Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/716754c9f228a4f63d8e6edb81e200d6f4ea0c3a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in 3D Gaussian splatting (2024, unknown): “Quantitative reconstruction results and sizes of 3D scenes after compression are shown in Table 2.”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, background): “In particular, Chen et al. [6], Luiten et al. [22], Wu et al. [45], Yang et al. [48, 49] directly manipulate NeRF fields.”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “To achieve real-time dynamic scene rendering, some studies [26, 40, 44] have extended the 3D Gaussian Splatting technique to dynamic scenes.”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “Recently, 3D Gaussian splatting [15] has emerged as an alternative 3D representation to NeRF, showcasing impressive quality and speed in 3D and 4D reconstruction tasks [15, 26, 48, 51, 52].”
A Survey on 3D Gaussian Splatting (2024, unknown): “In the point densification phase, 3D GS adaptively increases the density of Gaussians to better capture the details of a scene.”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, unknown): “We compare our method with state-of-the-art methods including D-NeRF [37], TiNeuVox-B [5], K-Planes [38], and 4D-GS [59].”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, methodology): “This approach has also been extended to model dynamic scenes [20, 44, 47, 48] with dense 6-DOF motion [20].”
GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting (2023, background): “Beyond the rendering superiorities, Gaussian splatting holds an explicit geometry scene struc-2 ture and appearance, benefiting from the exact modeling of scenes representation [47].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Several follow-up methods [26, 52, 53] extend it to dynamic 3D scenes by tracking of all dense scene elements [26] or deformation field modeling [49, 52].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “It is also applied to dynamic scene reconstruction [28, 51, 55, 56] and 3D content generation [12, 42, 59].”
Gaussian Splatting SLAM (2023, unknown): “Several works have applied 3D Gaussians and differentiable rendering to static scene capture [11, 38], and in particular more recent works utilise 3DGS and demonstrate superior results in vision tasks such as dynamic scene capture [15, 42, 44] and 3D generation [35, 45].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”