Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

By Man Luo, Yankai Zeng, Pratyay Banerjee, Chitta Baral. 2021

98 citations recorded by the source corpus.

Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models’ performance. To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and images and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experimental results show that a good retriever can significantly improve the reader’s performance on the OK-VQA challenge. The code and corpus are provided in https://github.com/luomancs/retriever_reader_for_okvqa.git.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Question Answering & Reading Comprehension, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Chenguang Zhu
3. Lu Yuan
4. Yejin Choi
5. Yichong Xu
6. Bill Byrne
7. Yonatan Bisk
8. Premkumar Natarajan

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Lu Yuan
3. Luc Van Gool
4. Dacheng Tao
5. Trevor Darrell
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Yi Yang

In Information Retrieval & Search

1. Lu Yuan
2. Weizhe Lin
3. Jianfeng Gao
4. Yichong Xu
5. Yonatan Bisk
6. Bill Byrne
7. Dongdong Chen
8. Alexander G. Hauptmann

In Question Answering & Reading Comprehension

1. Weizhe Lin
2. Jianfeng Gao
3. Bill Byrne
4. Chenguang Zhu

In Knowledge Graphs & Representation

1. Weizhe Lin
2. Jianfeng Gao
3. Bill Byrne
4. Chenguang Zhu

Citations in the corpus

• KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, methodology): “Further, external APIs, such as Google (Wu et al., 2021; Luo et al., 2021), Microsoft (Yang et al., 2021), and OCR (Luo et al., 2021; Wu et al., 2021) are used to enrich the associated knowledge.”
• Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “Second, a concurrent work, VRR [37], transforms the image into caption text and performs span-based question answering on a trimmed knowledge base using Google search engine.”
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “4 Visual Retriever-Reader [20] Google Search 39.”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “Prior work (Qu et al., 2021; Luo et al., 2021) has addressed this problem using “ Pseudo Relevance Labels ” which are based on whether a document a r X i v : 2210 .”