ConceptBert: Concept-Aware Representation for Visual Question Answering

By François Gardères, Maryam Ziaeefard, Baptiste Abeloos, Freddy Lécué. 2020

203 citations recorded by the source corpus.

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. A VQA model combines visual and textual features in order to answer questions grounded in an image. Current works in VQA focus on questions which are answerable by direct analysis of the question and image alone. We present a concept-aware algorithm, ConceptBert, for questions which require common sense, or basic factual knowledge from external structured content. Given an image and a question in natural language, ConceptBert requires visual elements of the image and a Knowledge Graph (KG) to infer the correct answer. We introduce a multi-modal representation which learns a joint Concept-Vision-Language embedding inspired by the popular BERT architecture. We exploit ConceptNet KG for encoding the common sense knowledge and evaluate our methodology on the Outside Knowledge-VQA (OK-VQA) and VQA datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Mohit Bansal
4. Lijuan Wang
5. Cordelia Schmid
6. Zicheng Liu
7. Yejin Choi
8. Trevor Darrell

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Trevor Darrell
4. Dacheng Tao
5. Jitendra Malik
6. Ming-Hsuan Yang
7. Andrew Zisserman
8. Xiaogang Wang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Zicheng Liu
5. Jianfeng Wang
6. Xiaowei Hu
7. Zhengyuan Yang
8. Devi Parikh

In Question Answering & Reading Comprehension

1. Weizhe Lin
2. Bill Byrne
3. Lijuan Wang
4. Jianfeng Gao
5. Xiaowei Hu
6. Zhengyuan Yang
7. Zicheng Liu
8. Chitta Baral

Citations in the corpus

• KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, methodology): “…(Marino et al., 2019) - 26.41 BAN+KG-AUG (Li et al., 2020b) Wikipedia+ConceptNet 26.71 MUTAN+AN (Marino et al., 2019) Wikipedia 27.84 ConceptBERT (Garderes et al., 2020) ConceptNet 33.66 KRISP (Marino et al., 2021) Wikipedia + ConceptNet 38.35 Visual Retriever-Reader (Luo et al., 2021) Google…”
• An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “To obtain such knowledge, existing methods (Zhu et al. 2020; Garderes et al. 2020; Marino et al. 2021; Wu et al. 2021) first retrieve external knowledge from multiple resources, such as Wikipedia articles and ConceptNet concepts (Speer, Chin, and Havasi 2017).”
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most early methods [20,23,29,40,58,71,76] use deep neural networks to understand images and retrieve relevant knowledge from explicit knowledge bases.”
• Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “Most existing work for OK-VQA rely on the pre-trained vision-language models as a major workhorse for question answering [12, 37, 39, 49, 61, 64].”
• Multi-Modal Answer Validation for Knowledge-Based VQA (2021, background): “Most recent KB-VQA systems (Gardères et al. 2020; Marino et al. 2021; Shevchenko et al. 2021) utilize multimodal transformers (Lu et al.”
• PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, unknown): “ConceptBERT [16] Feature ConceptNet 33.”
• Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory (2022, background): “One line of research uses explicit knowledge either from structured knowledge graphs [11, 30, 43] or unstructured text corpora [26, 27, 45].”
• Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021, methodology): “ConceptBert (Gardères et al., 2020) combines the BERT-pretrained model (De-vlin et al., 2019) with KG.”
• KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, methodology): “For improved generalization to open-domain VQA, [26, 52, 40, 80] embed the symbolic knowledge to an implicit embedding loosing the semantics of the symbols, but therefore are able to easily integrate the embedding with standard VQA approaches.”
• Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “KB-VQA can access both structured data, such as ConceptNet and other KGs (Narasimhan et al., 2018a; Garderes et al., 2020; Li et al., 2020a; Wu et al., 2022; Marino et al., 2021), as well as unstructured data such as Wikipedia passages (Wu et al., 2022; Gao et al., 2022; Gui et al., 2021).”