Visual Grounding Via Accumulated Attention

By Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, Mingkui Tan. 2020

205 citations recorded by the source corpus.

Visual grounding (VG) aims to locate the most relevant object or region in an image, based on a natural language query. Generally, it requires the machine to first understand the query, identify the key concepts in the image, and then locate the target object by specifying its bounding box. However, in many real-world visual grounding applications, we have to face with ambiguous queries and images with complicated scene structures. Identifying the target based on highly redundant and correlated information can be very challenging, and often leading to unsatisfactory performance. To tackle this, in this paper, we exploit an attention module for each kind of information to reduce internal redundancies. We then propose an accumulated attention (A-ATT) mechanism to reason among all the attention modules jointly. In this way, the relation among different kinds of information can be explicitly captured. Moreover, to improve the performance and robustness of our VG models, we additionally introduce some noises into the training procedure to bridge the distribution gap between the human-labeled training data and the real-world poor quality data. With this “noised” training strategy, we can further learn a bounding box regressor, which can be used to refine the bounding box of the target object. We evaluate the proposed methods on four popular datasets (namely ReferCOCO, ReferCOCO+, ReferCOCOg, and GuessWhat?!). The experimental results show that our methods significantly outperform all previous works on every dataset in terms of accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “e (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as crossmodal retrieval [21,61], visual question answering [25], captioning [56] or visual grounding [11,30]. Building on the recent advances in natural language processing [14,59], several works have ”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Many approaches provide explanations in the form of pixel importance, à la Grad-CAM [47, 65, 11, 41], some also providing textual explanations [41].”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “Prophet Attention: In prophet attention [165], the authors noticed that the conventional attention models are biased and cause deviated focus to the tasks, especially in sequence models such as image captioning [166], [167] and visual grounding [168], [169].”
Multi-Domain Multi-Task Rehearsal for Lifelong Learning (2020, background): “In contrast to static machine learning (He et al. 2016; Deng et al. 2018; Lyu et al. 2019; Lyu, Feng, and Wang 2020), Lifelong Learning (Ring 1997; Thrun 1998) is proposed to improve the self-learning ability of the machine that continually learns new knowledge.”
Visual Grounding with Transformers (2021, methodology): “The propose-and-rank methods [13, 12, 3, 4, 5, 14] first generate a set of candidate object proposals from the image by leveraging off-the-shelf detectors [15] or proposal generators, then score the candidates with respect to the language description and choose the top-ranked one.”
Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, background): “Attention mechanisms have become an indispensable component to build neural architectures for vision-language tasks, such as VQA [5, 17, 20, 28, 29, 42, 45, 46] and visual grounding [11, 43, 49], etc.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2018b) VGG BiLSTM VarContext SGD 3 (Deng et al., 2018) VGG LSTM AccumulateAtt SGD 7 (Zhuang et al.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “Early works on combining image and text modalities did not focus on pre-training for downstream tasks but instead on learning aligned representations for cross-modal retrieval[80,24], on encoder-decoder tasks like image captioning[79,39,17], and on joint prediction tasks like visual question answeri”
Open-Vocabulary DETR with Conditional Matching (2022, background): “Visual Grounding is another relevant research area where the problem is to ground a target object in one image using natural language input [5, 3].”