Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis

By Jonathon Luiten, Georgios Kopanas, Bastian Leibe, Deva Ramanan. International Conference on 3D Vision, 2023

923 citations recorded by the source corpus.

We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians’ motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.111. Project Website: dynamic3dgaussians.github.io

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8ba2c82fe675ede1d5c12fc4f97cf8ca3ebf1ca3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, neutral): “Generating novel views from a set of posed images has seen significant progress [25, 35, 50], with numerous extensions towards large scene reconstruction [3, 28, 62, 71], improved rendering quality [2, 21, 58], faster rendering speed [37, 58] and handling dynamic scenes [11, 32].”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, methodology): “This approach has also been extended to model dynamic scenes [20, 44, 47, 48] with dense 6-DOF motion [20].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, positive): “Recently, 3D Gaussian splatting [15] has emerged as an alternative 3D representation to NeRF, showcasing impressive quality and speed in 3D and 4D reconstruction tasks [15, 26, 48, 51, 52].”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “This method has been extended to capture dynamic scenes by building explicit correspondences across time steps [28]; however, they do not allow for animations of the reconstructed outputs.”
A Survey on 3D Gaussian Splatting (2024, neutral): “Table S3) according to our taxonomy.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
Recent advances in 3D Gaussian splatting (2024, positive): “Vector Quantization, a traditional compression method in signal processing, which involves clustering multi-dimensional data into a finite set of representations, is mainly utilized in Gaussians [40–44].”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “Alternatively, more specialized point trackers can be used [141– 148].”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Recently, 3D Gaussian splatting (Kerbl et al., 2023) has been proposed as an alternative 3D representation to NeRF, which has demonstrated impressive quality and speed in 3D reconstruction (Luiten et al., 2023).”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Rather than reconstructing the scene frame by frame [26], we condition the 3D Gaussians on time and jointly train a purely implicit deformation field with the learnable 3D Gaussians in canonical space.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Several follow-up methods [26, 52, 53] extend it to dynamic 3D scenes by tracking of all dense scene elements [26] or deformation field modeling [49, 52].”
Gaussian Splatting SLAM (2023, positive): “Several works have applied 3D Gaussians and differentiable rendering to static scene capture [11, 38], and in particular more recent works utilise 3DGS and demonstrate superior results in vision tasks such as dynamic scene capture [15, 42, 44] and 3D generation [35, 45].”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, methodology): “One direct and effective extension approach is to construct 3D Gaussians at each timestamp [1, 19] but the storage/memory cost will multiply especially for long input sequences.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “While this method excels with dense input views and has achieved success in various 3D tasks [23, 39, 47], its reconstruction with sparse view inputs remains an open problem.”
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, background): “…approach explicitly models continuous motion or deformation, presuming the dynamics of the scene result from the movement or deformation of particular static structures, like particle or canonical fields (Pumarola et al., 2020; Song et al., 2023; Abou-Chakra et al., 2022; Luiten et al., 2024).”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “It is also applied to dynamic scene reconstruction [28, 51, 55, 56] and 3D content generation [12, 42, 59].”
GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting (2023, background): “Fortunately, recent work [11, 15, 44] with 3D Gaussian representation and tile-based splatting techniques has shown great superiority in the efficiency of high-resolution image rendering.”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”