CRIS: CLIP-Driven Referring Image Segmentation

By Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yan Guo, Ming Gong, Tongliang Liu. Computer Vision and Pattern Recognition, 2021

540 citations recorded by the source corpus.

Referring image segmentation aims to segment a referent via a natural linguistic expression. Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmen-tation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/76a2b197b5427ffd1d3470c6d3ea026588eb5d0a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GRES: Generalized Referring Expression Segmentation (2023, methodology): “Most recently, some transformer-based methods [5, 6, 39, 44] are proposed and bring large performance gain compared to the CNN-based network.”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, background): “Many recent works have also con-ﬁrmed the observation and achieved impressive performance on open-vocabulary image recognition [1,36,37] and other downstream tasks [13,15,26,30].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “For example, 160 SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels, 161 traine with supervision v a images with g ound-truth semantic labels.”
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, background): “[14], have recently been applied to multiple vision tasks [57, 74, 6, 53], but rarely to the medical domain [16, 75].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “With the help of CLIP’s knowledge, CRIS [198] proposes vision-language decoding and contrastive learning for achieving text-to-pixel alignment.”
LERF: Language Embedded Radiance Fields (2023, methodology): “CRIS [38] and CLIPSeg [23] train a 2D image decoder to output a relevancy map based on the query CLIP embedding and the intermediate outputs of the pretrained CLIP image encoder.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Pre-trained Vision Language Models [19, 22, 36, 40] which connects the image representation with text embedding have achieved significant performance on various downstream tasks, like image retrieval [28], dense prediction [37], visual referring expression [43], visual question answering [21] and so”