Multi-Modal Answer Validation for Knowledge-Based VQA

By Jialin Wu, Jiasen Lu, Ashish Sabharwal, Roozbeh Mottaghi. 2021

186 citations recorded by the source corpus.

The problem of knowledge-based visual question answering involves answering questions that require external knowledge in addition to the content of the image. Such knowledge typically comes in various forms, including visual, textual, and commonsense knowledge. Using more knowledge sources increases the chance of retrieving more irrelevant or noisy facts, making it challenging to comprehend the facts and find the answer. To address this challenge, we propose Multi-modal Answer Validation using External knowledge (MAVEx), where the idea is to validate a set of promising answer candidates based on answer-specific knowledge retrieval. Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source. Our multi-modal setting is the first to leverage external visual knowledge (images searched using Google), in addition to textual knowledge in the form of Wikipedia sentences and ConceptNet concepts. Our experiments with OK-VQA, a challenging knowledge-based VQA dataset, demonstrate that MAVEx achieves new state-of-the-art results. Our code is available at https://github.com/jialinwu17/MAVEX

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Lijuan Wang
4. Zicheng Liu
5. Chunyuan Li
6. Mohit Bansal
7. Cordelia Schmid
8. Jianwei Yang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Dacheng Tao
4. Trevor Darrell
5. Ming-Hsuan Yang
6. Jianfeng Gao
7. Xiaogang Wang
8. Andrew Zisserman

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Zicheng Liu
5. Jianfeng Wang
6. Xiaowei Hu
7. Jianwei Yang
8. Chunyuan Li

In Question Answering & Reading Comprehension

1. Weizhe Lin
2. Bill Byrne
3. Jianwei Yang
4. Pratyay Banerjee
5. Zhe Gan
6. Yong Jae Lee
7. Man Luo
8. Trevor Darrell

Citations in the corpus

• An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “• Our method surpasses the state of the art (Wu et al. 2021) with no model fine-tuning.”
• Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory (2022, background): “One line of research uses explicit knowledge either from structured knowledge graphs [11, 30, 43] or unstructured text corpora [26, 27, 45].”
• Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
• K-LITE: Learning Transferable Visual Models with External Knowledge (2022, background): “To address the problem, acquiring external knowledge becomes an essential component for success [93, 60, 96].”
• PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “1 MAVEx [73] Feature Wikipedia + ConceptNet + Google Search 39.”
• Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021, background): “Span-Selector (Jain et al., 2021) extracts spans from the question to search most relative knowledge from Google, whereas MAVEx (Wu et al., 2021) votes among textual and visual knowledge from Wikipedia, ConceptNet, and Google Image.”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “MAVEx (Wu et al., 2022) uses multiple information sources (Google Images, Wikipedia sentences, and Con-ceptNet) to validate promising answer candidates.”
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “However, MAVEx [38] considers object regions as a kind of knowledge without using their visual representation to retrieve other knowledge, KRISP [22] utilizes object regions to learn implicit knowledge by a transformer-based model and retrieve external knowledge by the text symbols of these regions,”
• Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “Other work uses vision-language models as a knowledge-free VQA model first and later adjusts the predicted answers by fusion with knowledge graphs [39] or answer validation with knowledge text [61].”