Segment Anything
By Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo and 2 more. IEEE International Conference on Computer Vision, 2023
14,627 citations recorded by the source corpus.
We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive – often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at segment-anything.com to foster research into foundation models for computer vision. We recommend reading the full paper at: arxiv.org/abs/2304.02643.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/2304.02643
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Trevor Darrell
- Percy Liang
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Representation & Self-Supervised Learning
- Sergey Levine
- Andrea Vedaldi
- Jitendra Malik
- Chelsea Finn
- Trevor Darrell
- Antonio Torralba
- Andrew Zisserman
- Yoshua Bengio
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “Inspired by [9], we leverage GPT-4 to measure the quality of generated responses.”
- SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, background): “The recent advancements in interactive segmentation [25, 53] show an elegant solution to this issue.”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “LlamaGen Janus (Ours) LlamaGen SDXL SDXL User: Explain this meme.”
- Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “But the “anything mask labels” by original SAM [15] have no direct semantic language information.”
- GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, background): “…reconstruction (top, ∼ 0.1s), GRM works in synergy with other tools (e.g., text-to-multiview generation [46], image-to-multiview model [79], and 2D segmentation [45]) to enable text-to-3D (center top) and image-to-3D (center bottom) generation as well as real-world object reconstruction (bottom).”
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “It collects 62M diverse and informative images from eight public large-scale datasets, e.g ., SA-1B [27], Open Images [30], and BDD100K [81].”
- MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “3 of Appendix G.”
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Within the realm of the vision community, there is a burgeoning interest in the zero-shot and in-context learning abilities of foundation models, CLIP [47], SAM [35], Dinov2 [44].”
- RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, methodology): “Numerous foundation models have emerged in recent years, such as SimCLR [2], mean absolute error (MAE) [3], and segment anything model (SAM) [4] for computer vision, Bidirectional Encoder Representations from Transformers (BERT) [5] and generative pre-trained transformer (GPT) series [6], [7] for…”
- RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “The SAM model, trained on over a billion masks, demonstrates exceptional generalization capabilities, enabling it to segment any object in any image without additional training [33].”
- π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, positive): “…subtasks to accomplish (e.g., pick up the plate) and emitting low-level actions. abling broad generalization, particularly with recent advances that have enabled scalable learning systems in domains ranging from natural language processing [79, 21, 10, 78] to computer vision [34, 66, 35, 43].”
- Scaling Open-Vocabulary Object Detection (2023, methodology): “A similar approach was recently applied with good results to class-agnostic segmentation in the Segment Anything work [11].”
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation (2023, positive): “Very recently, the Segmentation Anything Model (SAM) [21] has gained massive attention as a powerful and general vision segmentation model capable of generating various and fine-grained segmentation masks conditioned by the user prompt.”
- A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Additionally, they can adapt to changing road conditions with a level of understanding that closely resembles human intuition.”
- AnyDoor: Zero-shot Object-level Image Customization (2023, neutral): “The segmentor model could be either automatic [26, 38] or interactive [11, 12, 32].”
- SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “We do not use existing datasets since they either fall short due to vocabulary limitations, or use polygon annotations [37] or compressed masks [38] for segmentation.”
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “The Segment Anything Model (SAM) [17] allows for both prompt-able and promptless zero-shot segmentation in 2D, without the need for specific task training.”
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “For general objects, we employ a state-of-the-art segmentation and tracking method, namely SAM-Track [11], to analyze the correctness of the video content that we are interested in. Leveraging the powerful segmentation model [29], we can easily obtain the objects and their attributes.”