Image Segmentation Using Text and Image Prompts

By Timo Lüddecke, Alexander S. Ecker. 2021

794 citations recorded by the source corpus.

Image segmentation is usually addressed by training a model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated. Finally, we find our system to adapt well to generalized queries involving affordances or properties. Code is available at https://eckerlab.org/code/CLIPSeg

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “CLIPSeg [55] extended the textual query to a visual query and showed superior performance not only on referring segmentation but also on semantic segmentation.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “An image prompt serves as a visual anchor for the to-be-generated video’s content and other elements such as characters, setting, and mood [90].”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, methodology): “The resulting model is suited for downstream tasks such as open-vocabulary image recognition [23], object detection [11], and image segmentation [29].”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “CLIPSeg [175] introduces a lightweight transformer decoder to extend CLIP for semantic segmentation.”
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, methodology): “Specifically, given an image containing a subject, we first feed the image and a category text of the subject to a text-prompted segmentation model CLIPSeg [23] with confidence thresholding.”
MaPLe: Multi-modal Prompt Learning (2022, methodology): “Many works have demonstrated better performance on downstream tasks by using tailored methods to adapt V-L models for few-shot image-recognition [9, 19, 46], object detection [8,10,27,34,44,50], and segmentation [5,22,26,33].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “From the task prompt view, CLIPSeg [270] builds a system to generate image segmentations based on arbitrary prompts at test time.”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, background): “, 2019), as well as the open-vocabulary CLIPSeg (Lüddecke and Ecker, 2022) trained on the PhraseCut dataset (Wu et al.”
LERF: Language Embedded Radiance Fields (2023, methodology): “CRIS [38] and CLIPSeg [23] train a 2D image decoder to output a relevancy map based on the query CLIP embedding and the intermediate outputs of the pretrained CLIP image encoder.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “CLIPSeg [147] exploits CLIP’s [42] generalization capabilities to achieve zero-shot and one-shot segmentation.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “Although some approaches have adapted foundation models like CLIP [24, 18, 55], these require additional training on labelled image-text data.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Additionally, they can adapt to changing road conditions with a level of understanding that closely resembles human intuition.”
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, background): “With the widespread success of large models in the field of language processing and understanding [15, 4, 64, 41], large-scale pre-trained visionlanguage models (VLM), e.”