RegionCLIP: Region-based Language-Image Pretraining

By Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Noel C. F. Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, Jianfeng Gao. 2021

921 citations recorded by the source corpus.

Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification in both zero-shot and transfer learning set-tings. However, we show that directly applying such mod-els to recognize image regions for object detection leads to unsatisfactory performance due to a major domain shift: CLIP was trained to match an image as a whole to a text de-scription, without capturing the fine-grained alignment be-tween image regions and text spans. To mitigate this issue, we propose a new method called RegionCLIP that signifi-cantly extends CLIP to learn region-level visual representations, thus enabling fine-grained alignment between image regions and textual concepts. Our method leverages a CLIP model to match image regions with template captions, and then pretrains our model to align these region-text pairs in the feature space. When transferring our pretrained model to the open-vocabulary object detection task, our method outperforms the state of the art by 3.8 AP50 and 2.2 AP for novel categories on COCO and LVIS datasets, respectively. Further, the learned region representations support zero-shot inference for object detection, showing promising results on both COCO and LVIS datasets. Our code is available at https://github.com/microsoft/RegionCLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, background): “To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [24, 14], with strong capabilities in open-world visual understanding such as classification [36, 18, 53, 50, 35], detection [26, 58, 29], segmentation [23, 59, 54] and captioning ”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Previous works have explored different ways of leveraging a large amount of image-text data for open-vocabulary detection or segmentation, such as distilling the visual-semantic representations from multi-modal foundation models [12, 56], designing fine-grained or augmented contrastive learning meth”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ViLD [13] Mask R-CNN [15] X sentence X partial label partial label RegionCLIP [62] Faster RCNN [39] X sentence X partial label partial label FindIt [21] Faster RCNN [39] A sentence X partial label fine-tune MDETR [18] DETR [2] A,C word fine-tune zero-shot fine-tune DQ-DETR [46] DETR [2] A,C word X z”
Generalized Decoding for Pixel, Image, and Language (2022, background): “VL pretrained models can be further extended to region-level tasks, such as phrase grounding and open-vocabulary object detection [26, 35, 58, 97], and unified frameworks that aim to Figure 2.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The success has quickly inspired many follow-up large-scale pre-training works [68, 72, 69, 36, 43, 20, 34, 74].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Many recent works aim to transfer the language capabilities of these models to object detection [12,26,45,46,20].”
Scaling Open-Vocabulary Object Detection (2023, methodology): “Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases or concepts [40, 33, 32].”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “RegionClip [153] ResNet50x4 Transformer 118k 29.”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, neutral): “6 Acknowledgement”
Exploring CLIP for Assessing the Look and Feel of Images (2022, background): “…and Hoshen 2021; Xu et al. 2022), image cap-tioning (Hessel et al. 2021; Mokady, Hertz, and Bermano 2021), view synthesis (Jain, Tancik, and Abbeel 2021), object detection (Gu et al. 2022; Zhong et al. 2022; Shi et al. 2022), and semantic segmentation (Rao et al. 2022; Zhou, Loy, and Dai 2022).”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “Standard single-car ga-rages usually have a width of about 10 to 12 feet and a depth of about 20 to 24 feet.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, neutral): “3 Region-CLIP [75] (tuned on non-rare LVIS) 32.”
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks (2022, background): “OVOD [15,25,40,59,61] usually leverages language models as it is impossible to pre-define all open-vocabulary categories.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, positive): “We choose Mask R-CNN per existing works (Gu et al., 2022; Zareian et al., 2021; Zhong et al., 2022).”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Recent development of image encoder architectures [96, 22, 71] and large-scale training [69, 12] have improved the ability and generalization of vision models, including zero-shot models [44, 52, 99, 103, 114, 72].”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, positive): “CLIP [51] and RegionCLIP [72] enable zero-shot classiﬁcation and detection by replacing the class labels with natural language supervision during training.”
Open-vocabulary Attribute Detection (2022, methodology): “[56] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Beyond classification, CLIP models also exhibit the flexibility to be integrated into a variety of vision-language applications, including detection [33], [34], segmentation [35], [36], captioning [37], [38], VQA [39], [40], and conditional image generation [41].”