Retrieval Augmented Visual Question Answering with Outside Knowledge

By Weizhe Lin, Bill Byrne. 2022

137 citations recorded by the source corpus.

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance.Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Peng Gao
  2. Hongsheng Li
  3. Renrui Zhang
  4. Yu Qiao
  5. Jiaming Han
  6. Pan Lu
  7. Xiangfei Hu
  8. Shilin Yan

In Computer Vision and Pattern Recognition

  1. Hongsheng Li
  2. Peng Gao
  3. Yu Qiao
  4. Renrui Zhang
  5. Pan Lu
  6. Xiaogang Wang
  7. Jiaming Han
  8. Luc Van Gool

In Multimodal Learning

  1. Peng Gao
  2. Hongsheng Li
  3. Yu Qiao
  4. Renrui Zhang
  5. Pan Lu
  6. Jiaming Han
  7. Xiaogang Wang

In Question Answering & Reading Comprehension

  1. Peng Gao
  2. Hongsheng Li
  3. Yu Qiao
  4. Pan Lu

Citations in the corpus