Segment Anything in High Quality

By Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu. Neural Information Processing Systems, 2023

655 citations recorded by the source corpus.

The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out of them are evaluated in a zero-shot transfer protocol. Our code and pretrained models are at https://github.com/SysCV/SAM-HQ.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/36fff28902cfb6c99c7c98f284639ad8e0133c44

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

SAM 2: Segment Anything in Images and Videos (2024, methodology): “SAM 2 (Hiera-B+) trained only on SA-1B outperforms SAM (ViT-H) on 1-click accuracy, and both SAM (ViT-H) and HQ-SAM (ViT-H) on 5-click accuracy while being 6x faster.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, methodology): “* means the results were tested by the SAM-HQ [24] team.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “While there has been remarkable progress in 2D scene understanding brought by SAM and its variants [12, 15, 58], their extension to 3D has been constrained.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “…our RepViT-SAM with Mobile-SAM [71] and the original SAM [33] with ViT-B image We then evaluate the performance of our RepViT-SAM on zero-shot edge detection using BSDS500 [2, 45], zero-shot instance segmentation using COCO [37], and segmentation in the wild benchmark (SegInW), following [31, 33].”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “2(d)], we have conducted an extensive review of the current advancements and derivatives in the SAM community [35], [37], [42].”
Personalize Segment Anything Model with One Shot (2023, background): “There are some concurrent works extending SAM for high-quality segmentation (Ke et al., 2023), faster inference speed (Zhao et al.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, methodology): “Therefore, we made an initial attempt by carefully assigning semantic segmentation labels to our unlabeled images with a combination of RAM [85] + GroundingDINO [37] + HQ-SAM [26] models.”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “These boxes are then accurately segmented using HQ-SAM [17].”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “Finally, we apply segmentation models [36] to refine these bounding boxes into precise masks.”