Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

By Ziyi Yang, Xinyu Gao, Wenming Zhou, Shaohui Jiao, Yuqing Zhang, Xiaogang Jin. Computer Vision and Pattern Recognition, 2023

924 citations recorded by the source corpus.

Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction. Nonetheless, cutting-edge dynamic neural rendering methods rely heavily on these implicit representations, which frequently struggle to capture the intricate details of objects in the scene. Furthermore, implicit methods have difficulty achieving real-time rendering in general dynamic scenes, limiting their use in a variety of tasks. To address the issues, we propose a deformable 3D Gaussians splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space with a deformation field to model monocular dynamic scenes. We also introduce an annealing smoothing training mechanism with no extra overhead, which can mitigate the impact of inaccurate poses on the smoothness of time interpolation tasks in real-world scenes. Through a differential Gaussian rasterizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed. Experiments show that our method outperforms existing methods significantly in terms of both rendering quality and speed, making it well-suited for tasks such as novel-view synthesis, time interpolation, and real-time rendering. Our code is available at https://github.com/ingra14m/Deformable-3D-Gaussians.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1ce81d64eefe5915c6ef9719915efa5f4079a6c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Several follow-up methods [26, 52, 53] extend it to dynamic 3D scenes by tracking of all dense scene elements [26] or deformation field modeling [49, 52].”
Recent advances in 3D Gaussian splatting (2024, positive): “One way is to use an MLP network to implicitly fit it, similar to the dynamic NeRF [48].”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, methodology): “This approach has also been extended to model dynamic scenes [20, 44, 47, 48] with dense 6-DOF motion [20].”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
A Survey on 3D Gaussian Splatting (2024, positive): “This method also mitigated semantic ambiguity and enhanced the precision of open-vocabulary querying by smoothing out semantic features across different views, guided by uncertainty values. ii ) Scene Understanding and Editing [24, 295, 332].”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “[43] presented a deformable 3D Gaussians Splatting method, which learned 3D Gaussians in canonical space and modeled the dynamic scenes with a deformation field.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, background): “Recently, there are several work on this topic that are concurrent to ours [37, 45, 59, 64, 93, 95, 97, 99, 100].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “It is also applied to dynamic scene reconstruction [28, 51, 55, 56] and 3D content generation [12, 42, 59].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, positive): “Recently, 3D Gaussian splatting [15] has emerged as an alternative 3D representation to NeRF, showcasing impressive quality and speed in 3D and 4D reconstruction tasks [15, 26, 48, 51, 52].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “This representation has been quickly adopted for various application, including image-or text-conditioned 3D [14,47,49] and 4D generation [52,74], avatar reconstruction [31,48,72,76,113], dynamic scene reconstruction [61,102,107,108], among others [11,21,97].”
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, background): “In particular, Chen et al. [6], Luiten et al. [22], Wu et al. [45], Yang et al. [48, 49] directly manipulate NeRF fields.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Meanwhile, Deformable 3DGS [40] is a recurrent work that introduces an MLP deformation network to model the motion of dynamic scenes.”