Retrieval Augmented Visual Question Answering with Outside Knowledge
By Weizhe Lin, Bill Byrne. 2022
137 citations recorded by the source corpus.
Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance.Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Multimodal Learning
In Question Answering & Reading Comprehension
Citations in the corpus
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “More experiments on larger LLaMA models (33B, 65B parameters), and diverse benchmarks (VQA v2 [10], OK-VQA [25], TVQA [19], and DocVQA [29]) will be conducted.”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [13] 50.5 KAT [16] 54.4 RA-VQA [32] 54.5 REVIVE [33] 58.0 PromptCap [21] 58.8 Z S PNP-VQA [52] 35.9 PICa [60] 43.3 BLIP-2 [30] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
- See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most early methods [20,23,29,40,58,71,76] use deep neural networks to understand images and retrieve relevant knowledge from explicit knowledge bases.”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “RA-VQA [339] identifies a limitation that the retrieval in previous work is trained separately from answer generation, and propose a joint training scheme that integrates differentiable retrieval with answer generation, enabling end-to-end training.”