SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes

By Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, Xiaojuan Qi. Computer Vision and Pattern Recognition, 2023

414 citations recorded by the source corpus.

Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. Recently, Gaussian splatting has emerged as a robust technique to represent static scenes and enable high-quality and real-time novel view synthesis. Building upon this technique, we propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians, respectively. Our key idea is to use sparse control points, significantly fewer in number than the Gaussians, to learn compact 6 DoF transformation bases, which can be locally interpolated through learned interpolation weights to yield the motion field of 3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF transformations for each control point, which reduces learning complexities, enhances learning abilities, and facilitates obtaining temporal and spatial coherent motion patterns. Then, we jointly learn the 3D Gaussians, the canonical space locations of control points, and the deformation MLP to reconstruct the appearance, geometry, and dynamics of 3D scenes. During learning, the location and number of control points are adaptively adjusted to accommodate varying motion complexities in different regions, and an ARAP loss following the principle of as rigid as possible is developed to enforce spatial continuity and local rigidity of learned motions. Finally, thanks to the explicit sparse motion representation and its decomposition from appearance, our method can enable user-controlled motion editing while retaining high-fidelity appearances. Extensive experiments demonstrate that our approach outperforms existing approaches on novel view synthesis with a high rendering speed and enables novel appearance-preserved motion editing applications.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b54efb1ef315ecb68571b99ae7489d1834da02ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Bo Dai
2. Linning Xu
3. Tao Lu
4. Mulin Yu

In Computer Graphics and Computer-Aided Design

1. Bo Dai
2. Linning Xu
3. Lihan Jiang
4. Tao Lu
5. Mulin Yu
6. Kerui Ren
7. Zhangkai Ni

In Computer Vision and Pattern Recognition

1. Bo Dai
2. Linning Xu
3. Lihan Jiang
4. Tao Lu
5. Mulin Yu
6. Kerui Ren
7. Zhangkai Ni
8. Luc Van Gool

Citations in the corpus

• Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, background): “Recently, there are several work on this topic that are concurrent to ours [35, 42, 54, 59, 85, 87, 89, 91, 92].”
• Recent advances in 3D Gaussian splatting (2024, neutral): “Quantitative reconstruction results and sizes of 3D scenes after compression are shown in Table 2.”
• Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, background): “…rapidly expanded 3D-GS into diverse downstream applications beyond static 3D reconstruction, sparking a surge of extended applications to 3D generative modeling [31]–[33], physical simulation [13], [34], dynamic modeling [35]–[37], SLAMs [38], [39], and autonomous driving scenes [10]–[12], etc.”
• A Survey on 3D Gaussian Splatting (2024, neutral): “These results indicate the effectiveness of introducing additional properties or structured information to model the deformation of Gaussians so as to model the scene dynamics.”