Cascade Reasoning Network for Text-based Visual Question Answering

By Fen Liu, Guanghui Xu, Qi Wu, Qing Du, Mingkui Tan. 2020

63 citations recorded by the source corpus.

We study the problem of text-based visual question answering (T-VQA) in this paper. Unlike general visual question answering (VQA) which only builds connections between questions and visual contents, T-VQA requires reading and reasoning over both texts and visual concepts that appear in images. Challenges in T-VQA mainly lie in three aspects: 1) It is difficult to understand the complex logic in questions and extract specific useful information from rich image contents to answer them; 2) The text-related questions are also related to visual concepts, but it is difficult to capture cross-modal relationships between the texts and the visual concepts; 3) If the OCR (optical character recognition) system fails to detect the target text, the training will be very difficult. To address these issues, we propose a novel Cascade Reasoning Network (CRN) that consists of a progressive attention module (PAM) and a multimodal reasoning graph (MRG) module. Specifically, the PAM regards the multimodal information fusion operation as a stepwise encoding process and uses the previous attention results to guide the next fusion process. The MRG aims to explicitly model the connections and interactions between texts and visual concepts. To alleviate the dependence on the OCR system, we introduce an auxiliary task to train the model with accurate supervision signals, thereby enhancing the reasoning ability of the model in question answering. Extensive experiments on three popular T-VQA datasets demonstrate the effectiveness of our method compared with SOTA methods. The source code is available at https://github.com/guanghuixu/CRN_tvqa.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Luc Van Gool
5. Dacheng Tao
6. Cordelia Schmid
7. Jianfeng Wang
8. Xiaogang Wang

In Natural Language Processing

1. Zhe Gan
2. Linjie Li
3. Lijuan Wang
4. Xiaowei Hu
5. Zicheng Liu
6. Jianfeng Wang
7. Zhengyuan Yang
8. Kevin Qinghong Lin

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Xiaowei Hu
6. Linjie Li
7. Zhengyuan Yang
8. Ce Liu

Citations in the corpus

• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “First, “TAP” outperforms the state of the art [20, 25, 16, 36, 19] by large margins, and significantly improves the non-TAP baseline “M4C†.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[71] Fen Liu, Guanghui Xu, Qi Wu, Qing Du, Wei Jia, and Mingkui Tan.”