REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering

By Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, Lu Yuan. 2022

115 citations recorded by the source corpus.

This paper revisits visual representation in knowledge-based visual question answering (VQA) and demonstrates that using regional information in a better way can significantly improve the performance. While visual representation is extensively studied in traditional VQA, it is under-explored in knowledge-based VQA even though these two tasks share the common spirit, i.e., rely on visual input to answer the question. Specifically, we observe that in most state-of-the-art knowledge-based VQA methods: 1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent. Based on these observations, we propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model. The key motivation is that object regions and inherent relationship are important for knowledge-based VQA. We perform extensive experiments on the standard OK-VQA dataset and achieve new state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous state-of-the-art method by a large margin (+3.6%). We also conduct detailed analysis and show the necessity of regional information in different framework components for knowledge-based VQA. Code is publicly available at https://github.com/yzleroy/REVIVE.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Cordelia Schmid

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [13] 50.5 KAT [16] 54.4 RA-VQA [32] 54.5 REVIVE [33] 58.0 PromptCap [21] 58.8 Z S PNP-VQA [52] 35.9 PICa [60] 43.3 BLIP-2 [30] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory (2022, methodology): “They utilize off-the-shelf image caption models to turn images into text, feed them into a language model, and use the generated text output as augmented knowledge [13, 23, 47].”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Recent methods [26,36,41,64] found that LLMs like GPT-3 could serve as a knowledge base and use the LLM to answer the question directly.”