R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering

By Pan Lu, Lei Ji, Nan Duan, M. Zhou, Jianyong Wang. 2018

84 citations recorded by the source corpus.

Recently, Visual Question Answering (VQA) has emerged as one of the most significant tasks in multimodal learning as it requires understanding both visual and textual modalities. Existing methods mainly rely on extracting image and question features to learn their joint feature embedding via multimodal fusion or attention mechanism. Some recent studies utilize external VQA-independent models to detect candidate entities or attributes in images, which serve as semantic knowledge complementary to the VQA task. However, these candidate entities or attributes might be unrelated to the VQA task and have limited semantic capacities. To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA. Specifically, we build up a Relation-VQA (R-VQA) dataset based on the Visual Genome dataset via a semantic similarity module, in which each data consists of an image, a corresponding question, a correct answer and a supporting relation fact. A well-defined relation detector is then adopted to predict visual question-related relation facts. We further propose a multi-step attention model composed of visual attention and semantic attention sequentially to extract related visual knowledge and semantic knowledge. We conduct comprehensive experiments on the two benchmark datasets, demonstrating that our model achieves state-of-the-art performance and verifying the benefit of considering visual relation facts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Linjie Li
3. Hongsheng Li
4. Jingjing Liu
5. Yu Cheng
6. Jianfeng Gao
7. Peng Gao
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Dacheng Tao
3. Hongsheng Li
4. Luc Van Gool
5. Zhe Gan
6. Yi Yang
7. Cordelia Schmid
8. Ming-Hsuan Yang

In Question Answering & Reading Comprehension

1. Peng Gao
2. Xiaogang Wang
3. Hongsheng Li
4. Jingjing Liu
5. Linjie Li
6. Yu Cheng
7. Zhe Gan
8. Song-Chun Zhu

Citations in the corpus

• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Inspired by the recent remarkable performance achieved for VQA [32, 31, 9, 8, 25, 7], in this paper, we further extensively benchmark S CIENCE QA with a wide range of attention-based [1, 32, 20, 8] and Transformer-based [29, 25, 26, 7] methods.”
• Question-Guided Hybrid Convolution for Visual Question Answering (2018, methodology): “Recently, CNN and RNN are jointly trained for learning feature representations for multi-modal tasks, including image captioning [3,4], text-to-image retrieval [5,33], and Visual Question Answering (VQA) [6,11,12,38].”
• Relation-Aware Graph Attention Network for Visual Question Answering (2019, methodology): “Most of these methods applied VQAindependent models to extract semantic knowledge from the image, while [34] built a Relation-VQA dataset and directly mined VQA-specific relation facts to feed additional semantic information to the model.”
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Early VQA approaches usually combine multi-modal inputs by applying attention mechanisms over image regions or question words [28, 39, 38, 12, 54, 11].”
• BERT Representations for Video Question Answering (2020, background): “Moreover, most of the efforts in VQA systems are mainly focused on extracting better visual representation from images [23, 6, 26], leaving the modelling of the semantic language contribution to standard recurrent neural networks (RNN).”
• Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, background): “Several knowledge-based VQA datasets are proposed, each providing its own knowledge bases and ground-truth supporting fact [35, 48, 57].”