FindIt: Generalized Localization with Natural Language Queries

By Weicheng Kuo, Fred Bertsch, A. Piergiovanni, Mohammad Taghi Saffar, Anelia Angelova. 2022

18 citations recorded by the source corpus.

We propose FindIt, a simple and versatile framework that unifies a variety of visual grounding and localization tasks including referring expression comprehension, text-based localization, and object detection. Key to our architecture is an efficient multi-scale fusion module that unifies the disparate localization requirements across the tasks. In addition, we discover that a standard object detector is surprisingly effective in unifying these tasks without a need for task-specific design, losses, or pre-computed detections. Our end-to-end trainable framework responds flexibly and accurately to a wide range of referring expression, localization or detection queries for zero, one, or multiple objects. Jointly trained on these tasks, FindIt outperforms the state of the art on both referring expression and text-based localization, and shows competitive performance on object detection. Finally, FindIt generalizes better to out-of-distribution data and novel categories compared to strong single-task baselines. All of these are accomplished by a single, unified and efficient model. The code will be released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Feng Li
2. Hao Zhang
3. Lei Zhang
4. Jianwei Yang
5. Hang Su
6. Jun-Juan Zhu
7. Shilong Liu
8. Kevin Qinghong Lin

In Computer Vision and Pattern Recognition

1. Lei Zhang
2. Jianwei Yang
3. Feng Li
4. Hao Zhang
5. Hang Su
6. Shilong Liu
7. Jianfeng Gao
8. Jun-Juan Zhu

In Natural Language Processing

1. Lei Zhang
2. Jianwei Yang
3. Kevin Qinghong Lin
4. Marco Pavone
5. Jeannette Bohg
6. Zhaoyang Zeng
7. Hang Su
8. Feng Li

Citations in the corpus

• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ViLD [13] Mask R-CNN [15] X sentence X partial label partial label RegionCLIP [62] Faster RCNN [39] X sentence X partial label partial label FindIt [21] Faster RCNN [39] A sentence X partial label fine-tune MDETR [18] DETR [2] A,C word fine-tune zero-shot fine-tune DQ-DETR [46] DETR [2] A,C word X z”
• Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Suitable examples could be scene graph generation methods [43, 27] or forms of text-based scene description [22, 35, 58].”