Segment Everything Everywhere All at Once

By Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, Yong Jae Lee. Neural Information Processing Systems, 2023

838 citations recorded by the source corpus.

In this work, we present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image, as shown in Fig.1. In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder to image features; and iv) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it a readily universal image segmentation interface.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0819c1e60c13b9797f937282d06b54d252d9d6ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Diversifying SAM’s Prompting Mechanism: Zou et al. [148] proposetheSEEMmodelasauniversalinterfaceforthesegmen-tationofeverything,everywhere,withmulti-modalprompts.”
Segment Anything in Medical Images (2023, background): “Recently, segmentation foundation models have seen tremendous advancements in the ﬁeld of natural image segmentation [26][51][53], enabling accurate and eﬃcient segmentation of objects in a fully automatic or interactive way.”
Segment Anything in High Quality (2023, methodology): “For vision-based foundation models [21, 42, 56], prompt engineering [54, 44, 48, 55] that freezes the pre-trained model is first explored in vision-language models, such as CLIP [35].”
Segment anything in medical images (2023, neutral): “However, their application to medical image segmentation has been challenging due to the substantial domain gap [6] (Supplementary Related work).”
Personalize Segment Anything Model with One Shot (2023, methodology): “, 2022), SEEM (Zou et al., 2023), and SegGPT (Wang et al.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “In addition, several works [349], [350], [351] achieve better results.”
Evaluating Object Hallucination in Large Vision-Language Models (2023, methodology): “To validate it, we adopt SEEM (Zou et al., 2023) to annotate images from three datasets ( i.e., MSCOCO, A-OKVQA (Schwenk et al., 2022) and GQA (Hudson and Manning, 2019)) and build POPE based on the segmentation results.”
Ferret: Refer and Ground Anything Anywhere at Any Granularity (2023, methodology): “We ablate the effectiveness of the spatial-aware visual sampler by replacing it with the visual sampler in SEEM (Zou et al., 2023), where they average the features of all the sampled points as the region feature.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Currently, research on foundational segmentation and detection models has become a popular area of focus [33], [49], [50], [51], [52], [53].”