Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering

By Feng Gao, Qing Ping, Govind Thattai, Aishwarya N. Reganti, Yingting Wu, Premkumar Natarajan. 2022

93 citations recorded by the source corpus.

Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge. In this paper, we call for an alternative paradigm for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the natural language space. This paradigm takes advantage of the sheer volume of gigantic knowledge bases and the richness of pretrained language models. A Transform-Retrieve-Generate framework (TRiG) framework is proposed11The code of this work will be made public., which can be plug-and-played with alternative image-to-text models and textual knowledge bases. Experimental results show that our TRiG framework outperforms all state-of-the-art supervised methods by at least 11.1 % absolute margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Siyuan Huang
2. Zilong Zheng
3. Xiaojian Ma
4. Song-Chun Zhu
5. Weizhe Lin
6. Qing Li
7. Yitao Liang
8. Silong Yong

In Computer Vision and Pattern Recognition

1. Siyuan Huang
2. Zilong Zheng
3. Qing Li
4. Bill Byrne
5. Song-Chun Zhu
6. Weizhe Lin
7. Xiaojian Ma
8. Yitao Liang

In Multimodal Learning

1. Xiaojian Ma
2. Song-Chun Zhu
3. Weizhe Lin
4. Siyuan Huang
5. Qing Li
6. Yitao Liang
7. Silong Yong
8. Bill Byrne

In Question Answering & Reading Comprehension

1. Song-Chun Zhu
2. Weizhe Lin
3. Bill Byrne

Citations in the corpus

• ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [13] 50.5 KAT [16] 54.4 RA-VQA [32] 54.5 REVIVE [33] 58.0 PromptCap [21] 58.8 Z S PNP-VQA [52] 35.9 PICa [60] 43.3 BLIP-2 [30] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Large pre-trained models have also been used in reasoning over vision and language [21, 22, 32, 61, 69].”
• SQA3D: Situated Question Answering in 3D Scenes (2022, methodology): “Following prior practices that apply GPT-3 to VQA (Changpinyo et al., 2022; Gao et al., 2022), we propose to convert the 3D scene into text using an emerging technique called 3D captioning (Chen et al., 2021).”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “TRiG (Gao et al., 2022) shows that it is feasible to transform images into textual features for VQA.”