A Real-Time Cross-Modality Correlation Filtering Method for Referring Expression Comprehension

By Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, Bo Li. 2019

246 citations recorded by the source corpus.

Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Leonid Sigal
2. Muchen Li
3. Zhengyuan Yang
4. Lei Zhang
5. Jiebo Luo
6. Anelia Angelova
7. Hang Su
8. Qi Wu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Yi Yang
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Lei Zhang
7. Trevor Darrell
8. Jitendra Malik

Citations in the corpus

• GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
• CentripetalNet: Pursuing High-Quality Keypoint Pairs for Object Detection (2020, background): “With the development of deep learning, object detection has achieved great progress [10, 9, 30, 27, 25, 21, 11, 22, 1, 17, 34, 33, 36, 20, 19, 16, 39].”
• Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Following the previous works [8, 27, 51], we use Acc @0.5 to evaluate the performance of the network.”
• TransVG: End-to-End Visual Grounding with Transformers (2021, methodology): “RCCF [27] formulates the visual grounding problem as a correlation filtering process [4, 17], and picks the peak value of the correlation heatmap as the center of target objects.”
• DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “Most bottom-up solutions (Chen et al. 2018; Liao et al. 2019; Miao et al. 2022; Yang et al. 2019, 2020) fuse multi-modality features and output the final boxes directly.”
• Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, background): “Recently, several works [47,5,37,20,28] propose a different paradigm of onestage visual grounding.”
• Confidence-aware Non-repetitive Multimodal Transformers for TextCaps (2020, unknown): “In recent years, many works have focused on vision or language tasks (Zheng et al. 2019; Gao et al. 2020; Liao et al. 2020).”
• Visual Grounding with Transformers (2021, methodology): “The proposal-free methods [7, 9, 16, 10] focus on localizing the referred object directly without proposals.”
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “[26] utilizes CenterNet [11] to perform correlation filtering for region center localization.”
• FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “One-stage [7,44,89,13,55] and two-stage [94,91,27,83,87,50] methods have been popular for these tasks.”
• Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, background): “There has been growing interest in combining computer vision [31, 13, 9, 23, 15, 14, 29, 3, 4] and natural language processing [8, 45] techniques to perform vision-language cross-modal tasks [30, 38, 21, 22].”