Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

By Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan and 7 more. arXiv.org, 2024

1,209 citations recorded by the source corpus.

We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9cd6c6d85de6180dd92ba43e685663067cf3ab7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yinghao Xu
2. Ceyuan Yang
3. Bo Dai
4. Gordon Wetzstein
5. Yuwei Guo
6. Hongsheng Li
7. Hao He
8. Luc Van Gool

Citations in the corpus

• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “We first utilize the Grounded-SAM-2 (Ren et al., 2024) to segment the main object in a video.”
• DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “The sky masks help in addressing the ill-defined depth of the sky and can be obtained from semantic segmentation models [5, 13].”