Segment Anything Model for medical image analysis: an experimental study

By Maciej A. Mazurowski, Haoyu Dong, Han Gu, Jichen Yang, Nicholas Konz. Medical Image Anal., 2023

813 citations recorded by the source corpus.

Training segmentation models for medical images continues to be challenging due to the limited availability of data annotations. Segment Anything Model (SAM) is a foundation model trained on over 1 billion annotations, predominantly for natural images, that is intended to segment user-defined objects of interest in an interactive manner. While the model performance on natural images is impressive, medical image domains pose their own set of challenges. Here, we perform an extensive evaluation of SAM’s ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies. In our experiments, we generated point and box prompts for SAM using a standard method that simulates interactive segmentation. We report the following findings: (1) SAM’s performance based on single prompts highly varies depending on the dataset and the task, from IoU=0.1135 for spine MRI to IoU=0.8650 for hip X-ray. (2) Segmentation performance appears to be better for well-circumscribed objects with prompts with less ambiguity such as the segmentation of organs in computed tomography and poorer in various other scenarios such as the segmentation of brain tumors. (3) SAM performs notably better with box prompts than with point prompts. (4) SAM outperforms similar methods RITM, SimpleClick, and FocalClick in almost all single-point prompt settings. (5) When multiple-point prompts are provided iteratively, SAM’s performance generally improves only slightly while other methods’ performance improves to the level that surpasses SAM’s point-based performance. We also provide several illustrations for SAM’s performance on all tested datasets, iterative segmentation, and SAM’s behavior given prompt ambiguity. We conclude that SAM shows impressive zero-shot segmentation performance for certain medical imaging datasets, but moderate to poor performance for others. SAM has the potential to make a significant impact in automated medical image segmentation in medical imaging, but appropriate care needs to be applied when using it. Code for evaluation SAM is made publicly available at https://github.com/mazurowski-lab/segment-anything-medical-evaluation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

View Paper: https://www.semanticscholar.org/paper/c6ac64877b2bcac2389be4d91a62dea8c6f73844

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions (2024, background): “Similarly, Mazurowski et al. [11] conduct an extensive evaluation of SAM on a collection of 11 medical image segmentation datasets of different modalities by generating point prompts to simulate interactive segmentation.”
Segment anything in medical images (2023, positive): “These results highlight the potential of MedSAM as a powerful tool for medical image segmentation.”
LangSplat: 3D Language Gaussian Splatting (2023, background): “[38] introduced [48], super-resolution [25], image matting [45], object tracking [11, 42], medical image segmentation [29], image editing [12], and so on.”
SAM 2: Segment Anything in Images and Videos (2024, background): “The success of SAM led to its adoption in a wide range of applications, such as medical imaging (Ma et al., 2024; Deng et al., 2023; Mazurowski et al., 2023; Wu et al., 2023a), remote sensing (Chen et al., 2024; Ren et al., 2024), motion segmentation (Xie et al., 2024), and camouflaged object…”
Segment Anything in Medical Images (2023, background): “Several studies have shown that SAM could fail on typical medical image segmentation tasks [13], [20], [18], [39], [52], [37], [35] and other challenging scenarios [11],[46],[22],[23], especially when the targets have weak boundaries.”
On the Challenges and Perspectives of Foundation Models for Medical Image Analysis (2023, background): “However, its out-of-the-box performance on complex medical tasks such as pancreas, spine or cell nuclei segmentation is not satisfactory He et al. [2023], Roy et al. [2023], Deng et al. [2023], Mazurowski et al. [2023], Shi et al. [2023].”