GroupViT: Semantic Segmentation Emerges from Text Supervision

By Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, X. Wang. 2022

748 citations recorded by the source corpus.

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. We open-source our code at https://github.com/NVlabs/GroupViT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Apart from using foundation models, DenseCLIP [41] and GroupViT [46] show that fine-tuning from a foundation model or training from scratch can also yield superior zero-shot performance.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “[81] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “We tokenize and then encode the input text list (Tpad) using a 6-layer transformer text encoder [49, 57] to obtain a set of Ntext embeddings.”
LViT: Language Meets Vision Transformer in Medical Image Segmentation (2022, background): “Subsequently, there is a rich line of works on image segmentation [21], [27]–[31] that have begun to use text information to improve the segmentation capabilities of models.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, background): “GroupViT [38] groups segmentation masks directly from text supervision.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “Segmentation datasets for evaluation: Similar to recent works [17, 56] on zero-shot semantic segmentation, we use the following datasets for evaluation: a) Pascal VOC [16] (PV-20): 20 foreground classes with 1449 validation images, b) Pascal Context [36] (PC-59): 59 classes with 5k validation images”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “Many recent efforts [17,18,28,30,33,37,44,49,59,61,63] attempt to correlate the dense image features with the embedding from large language models.”
Segment Everything Everywhere All at Once (2023, methodology): “Many open-vocabulary segmentation models have been proposed recently [24, 13, 18, 10, 43, 51], which employ large pretrained vision-language models like CLIP [20] to transfer visual-semantic knowledge by freeze or fine-tune their weights.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “As CLIP gained popularity for image classification, numerous new approaches [18, 20, 23, 29, 36, 39, 41, 44, 53, 65, 68, 69] have emerged to tackle open-vocabulary image semantic segmentation.”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Meanwhile, several studies [45], [46], [67], [71], [129], [130], [131] model local fine-grained vision-language correlation via region-word matching, aiming for better dense predictions in object detection and semantic segmentation.”
Long-CLIP: Unlocking the Long-Text Capability of CLIP (2024, methodology): “As CLIP has strong zero-shot generalization abilities, it has been successfully utilized in detection [7,13], segmentation [12,31], video understanding [16,30] and most commonly, image generating [4,6,21,28].”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “There have also been recent methods [6, 39, 49, 63, 68] to explore the role of dynamic token within the transformer framework.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “Compared with GroupVit, OpenSeg learns visual grouping with class-agnostic segmentation, and has a superior performance.”
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, neutral): “Shortly after its inception, CLIP has found its way to (open-vocabulary) semantic segmentation [23, 68] and object detection [15, 26].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Meanwhile, several works [311], [312] explore the text-only supervision for semantic segmentation.”
k-means Mask Transformer (2022, methodology): “Finally, we note that several recent works [73,102,106,113] revisited the relationship be-tween query and key in the attention operation.”