Open-Vocabulary Image Segmentation

By Golnaz Ghiasi, Xiuye Gu, Yin Cui, Tsung-Yi Lin. 2021

97 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Many open-vocabulary segmentation models [7, 11, 17, 26] leverages large pre-trained vision-language models ( e.g. , CLIP [19] or ALIGN [40]) to distill or transfer the visual-semantic knowledge.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Furthermore, we reformulate the mask classification in segmentation into a mask-text matching problem between O s and the textual embeddings of prompted textual concepts similar to [24,79].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “Among these works, OpenSeg [29] also learns with class agnostic mask annotations to generate mask proposals, while our method does not require any mask annotations.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Beyond classification, CLIP models also exhibit the flexibility to be integrated into a variety of vision-language applications, including detection [33], [34], segmentation [35], [36], captioning [37], [38], VQA [39], [40], and conditional image generation [41].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “This allows us to leverage the backbone and pretraining of vision-language models in learning robotic policies, transferring some of their generalization, semantic understanding, and reasoning to robotic control.”
Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, methodology): “…regions” method in [8]; 2) MaskCLIP also avoids the challenging student-teacher distillation processes such as OVR-CNN [35] and ViLD [8] that face the problem of having a limited number of teacher objects to train; 3) MaskCLIP also learns to reﬁne masks beyond simple pooling in e.g. OpenSeg [7].”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “There are two concurrent work [30,16] try to utilize the vision-language pre-training model in semantic segmentation.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, methodology): “Our best model achieves 29.6% mIoU on A-150, which is +8.5% than the state-of-the-art OpenSeg [16] under the same setting.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “Segmentation datasets for evaluation: Similar to recent works [17, 56] on zero-shot semantic segmentation, we use the following datasets for evaluation: a) Pascal VOC [16] (PV-20): 20 foreground classes with 1449 validation images, b) Pascal Context [36] (PC-59): 59 classes with 5k validation images”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “For E 2D , we experiment with two pretrained image segmentation models OpenSeg [17] and LSeg [30].”
LERF: Language Embedded Radiance Fields (2023, methodology): “OpenSeg [14] simultaneously learns a mask prediction model while predicting the text embeddings for each mask, while ViLD [16] directly uses CLIP [28] to classify 2D regions from class-agnostic mask proposal networks.”
Segment Everything Everywhere All at Once (2023, methodology): “Many open-vocabulary segmentation models have been proposed recently [24, 13, 18, 10, 43, 51], which employ large pretrained vision-language models like CLIP [20] to transfer visual-semantic knowledge by freeze or fine-tune their weights.”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “We use the same prompt engineering strategy as in [8] to create an ensemble of text prompts for each test category and predict the category with the maximum probability.”
Detecting Twenty-thousand Classes using Image-level Supervision (2022, unknown): “ViLD [17], OpenSeg [16] and langSeg [29] upgrade the language embedding to CLIP [42].”