Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

By Dat T. Huynh, Jason Kuen, Zhe-nan Lin, Jiuxiang Gu, Ehsan Elhamifar. 2021

112 citations recorded by the source corpus.

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixelwise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1 % on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.11Code is available at https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Chunyuan Li
4. Lijuan Wang
5. Jianwei Yang
6. Linjie Li
7. Jianfeng Wang
8. Lu Yuan

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Jianwei Yang
4. Zhe Gan
5. Lei Zhang
6. Dacheng Tao
7. Chunyuan Li
8. Yong Jae Lee

In Multimodal Learning

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Chunyuan Li
5. Jianfeng Wang
6. Linjie Li
7. Jianwei Yang
8. Yong Jae Lee

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Many open-vocabulary segmentation models [7, 11, 17, 26] leverages large pretrained vision-language models (e.”
• Generalized Decoding for Pixel, Image, and Language (2022, background): “Prominent examples include LSeg [41], OpenSeg [25], and [33].”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For example, PB-OVD [189] trains object detectors with VLM-predicted pseudo bounding boxes while XPM [194] introduces a robust cross-modal pseudo-labeling strategy that employs VLM-generated pseudo masks for open-vocabulary instance segmentation.”
• Segment Everything Everywhere All at Once (2023, methodology): “Many open-vocabulary segmentation models have been proposed recently [24, 13, 18, 10, 43, 51], which employ large pretrained vision-language models like CLIP [20] to transfer visual-semantic knowledge by freeze or fine-tune their weights.”