Gaussian Grouping: Segment and Edit Anything in 3D Scenes

By Mingqiao Ye, Martin Danelljan, Fisher Yu, Lei Ke. European Conference on Computer Vision, 2023

475 citations recorded by the source corpus.

The recent Gaussian Splatting achieves high-quality and real-time novel-view synthesis of the 3D scenes. However, it is solely concentrated on the appearance and geometry modeling, while lacking in fine-grained object-level scene understanding. To address this issue, we propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. We augment each Gaussian with a compact Identity Encoding, allowing the Gaussians to be grouped according to their object instance or stuff membership in the 3D scene. Instead of resorting to expensive 3D labels, we supervise the Identity Encodings during the differentiable rendering by leveraging the 2D mask predictions by Segment Anything Model (SAM), along with introduced 3D spatial consistency regularization. Compared to the implicit NeRF representation, we show that the discrete and grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency. Based on Gaussian Grouping, we further propose a local Gaussian Editing scheme, which shows efficacy in versatile scene editing applications, including 3D object removal, inpainting, colorization, style transfer and scene recomposition. Our code and models are at https://github.com/lkeab/gaussian-grouping.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5f7867681c6287cc4f4e7fb5260af8067b52b3d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yin Yang
2. Zeshun Zong
3. Xuan Li
4. Tianyi Xie
5. Yuxing Qiu
6. Yutao Feng
7. Chenfanfu Jiang

Citations in the corpus

• PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, methodology): “Automatic parameter assignment could be derived from videos by combining GS segmentation [3, 50] with a differentiable MPM simulator.”
• A Survey on 3D Gaussian Splatting (2024, unknown): “Table S3) according to our taxonomy.”
• Recent advances in 3D Gaussian splatting (2024, unknown): “We have summarized the editing on 3DGS into three categories: geometry editing, appearance editing, and physical simulation.”