SAM-Adapter: Adapting Segment Anything in Underperformed Scenes

By Tianrun Chen, Lanyun Zhu, Chao Ding, Runlong Cao, Yan Wang, Zejian Li, Lingyun Sun, Ying-Dong Zang, Papa Mao. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023

359 citations recorded by the source corpus.

The emergence of large models, also known as foundation models, has brought significant advancements to AI research. One such model is Segment Anything (SAM), which is designed for image segmentation tasks. However, as with other foundation models, our experimental findings suggest that SAM may fail or perform poorly in certain segmentation tasks, such as shadow detection and camouflaged object detection (concealed object detection). This study first paves the way for applying the large pre-trained image segmentation model SAM to these downstream tasks, even in situations where SAM performs poorly. Rather than fine-tuning the SAM network, we propose SAM-Adapter, which incorporates domain-specific information or visual prompts into the segmentation network by using simple yet effective adapters. By integrating task-specific knowledge with general knowledge learnt by the large model, SAM-Adapter can significantly elevate the performance of SAM in challenging tasks as shown in extensive experiments. We can even outperform task-specific network models and achieve state-of-the-art performance in the task we tested: camouflaged object detection, shadow detection. Our code of adapting SAM in downstream applications have been released publicly at https://github.com/tianrun-chen/SAM-Adapter-PyTorch/ and has benefited many researchers. We believe our work opens up opportunities for utilizing SAM in downstream tasks, with potential applications in various fields, including medical image processing, agriculture, remote sensing, and more.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/20ace2e2d358098b284c39ff350934f904d8d6d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, background): “…has shown remarkable zero-shot transfer performance and high versatility for many vision tasks including a variety of segmentation application[7, 8, 10, 17], in-painting[67], image restoration[29], image editing[21], image shadow removal[69], object tracking[14, 65], and 3D object…”