X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks

By Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, Stefano Soatto. 2022

52 citations recorded by the source corpus.

In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image. To address these tasks, we propose X-DETR, whose architecture has three major components: an object detector, a language encoder, and vision-language alignment. The vision and language streams are independent until the end and they are aligned using an efficient dot-product operation. The whole network is trained end-to-end, such that the detector is optimized for the vision-language tasks instead of an off-the-shelf component. To overcome the limited size of paired object-language annotations, we leverage other weak types of supervision to expand the knowledge coverage. This simple yet effective architecture of X-DETR shows good accuracy and fast speeds for multiple instance-wise vision-language tasks, e.g., 16.4 AP on LVIS detection of 1.2K categories at ~20 frames per second without using any LVIS annotation during training.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Dacheng Tao
4. Zhe Gan
5. Yong Jae Lee
6. Jianwei Yang
7. Xiaogang Wang
8. Cordelia Schmid

In Multimodal Learning

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Chunyuan Li
5. Linjie Li
6. Jianfeng Wang
7. Jianwei Yang
8. Yong Jae Lee

Citations in the corpus

• Generalized Decoding for Pixel, Image, and Language (2022, background): “combine image-text pairs with region-level data have also been proposed [4, 20, 45, 87, 95].”
• Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, background): “The deep learning field is moving rapidly towards open-world/zero-shot settings [1] where computer vision tasks such as classification, object detection, semantic labeling, and image retrieval [1, 10, 35, 10, 14, 19, 2] perform recognition and detection for categories of custom specification beyond ”
• Transformer-Based Visual Segmentation: A Survey (2023, unknown): “X-DETR[238]proposesaneffective architectureforinstance-wisevision-languagetasksviausing dot-producttoalignvisionandlanguage.”
• DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, background): “Recently, GLIP [30] and X-DETR [2] try to align region and language features using a dot-product operation and can be trained end-to-end on both grounding data and detection data.”