Coarse-to-Fine Reasoning for Visual Question Answering
By Binh X. Nguyen, Tuong Khanh Long Do, Huy Tran, Erman Tjiputra, Quang D. Tran, Anh Gia-Tuan Nguyen. 2021
48 citations recorded by the source corpus.
Bridging the semantic gap between image and question is an important step to improve the accuracy of the Visual Question Answering (VQA) task. However, most of the existing VQA methods focus on attention mechanisms or visual relations for reasoning the answer, while the features at different semantic levels are not fully utilized. In this paper, we present a new reasoning framework to fill the gap between visual features and semantic clues in the VQA task. Our method first extracts the features and predicates from the image and question. We then propose a new reasoning framework to effectively jointly learn these features and predicates in a coarse-to-fine manner. The intensively experimental results on three large-scale VQA datasets show that our proposed approach achieves superior accuracy comparing with other state-of-the-art methods. Furthermore, our reasoning framework also provides an explainable way to understand the decision of the deep neural network when predicting the answer. Our source codes can be found at: https://github.com/aioz-ai/CFR_VQA
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Constantin Eichenberg
- Anette Frank
- Carl Vondrick
- Samuel Weinbach
- Sid Black
- D'idac Sur'is
- Sachit Menon
- Letitia Parcalabescu
In Computer Vision and Pattern Recognition
- D'idac Sur'is
- Sachit Menon
- Anette Frank
- Carl Vondrick
- Constantin Eichenberg
- Samuel Weinbach
- Sid Black
- Letitia Parcalabescu
In Question Answering & Reading Comprehension
Citations in the corpus
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “SOTA model SimVLM [55] PICa [58] CFR [33] Pythia [46] SimVLM SimVLM VIVO [22] SimVLM OSCAR [29]”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “By composing end-to-end modules into programs, ViperGPT brings the System 2 capability of sequential processing to deep learning [4].”