Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering

By Wei Han, Hantao Huang, Tao Han. 2020

55 citations recorded by the source corpus.

Image text carries essential information to understand the scene and perform reasoning. Text-based visual question answering (text VQA) task focuses on visual questions that require reading text in images. Existing text VQA systems generate an answer by selecting from optical character recognition (OCR) texts or a fixed vocabulary. Positional information of text is underused and there is a lack of evidence for the generated answer. As such, this paper proposes a localization-aware answer prediction network (LaAP-Net) to address this challenge. Our LaAP-Net not only generates the answer to the question but also predicts a bounding box as evidence of the generated answer. Moreover, a context-enriched OCR representation (COR) for multimodal fusion is proposed to facilitate the localization task. Our proposed LaAP-Net outperforms existing approaches on three benchmark datasets for the text VQA task by a noticeable margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Zicheng Liu
5. Jianfeng Wang
6. Xiaowei Hu
7. Zhengyuan Yang
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Luc Van Gool
3. Zicheng Liu
4. Lijuan Wang
5. Dacheng Tao
6. Cordelia Schmid
7. Chunhua Shen
8. Linjie Li

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Jianfeng Wang
5. Zicheng Liu
6. Zhengyuan Yang
7. Xiaowei Hu
8. Qi Wu

In Question Answering & Reading Comprehension

1. Anton van den Hengel
2. Qi Wu
3. Peng Wang
4. Zhe Gan
5. Jianfeng Wang
6. Zicheng Liu
7. Xiaowei Hu
8. Zhengyuan Yang

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[36] Wei Han, Hantao Huang, and Tao Han.”
• Structured Multimodal Attentions for TextVQA (2020, background): “Following on M4C, the LaAP [21] further predicts the OCR position as the evidence for final predicted answer, SA-M4C [22] replaced the original self-attention layers with their novel spatially aware selfattention layers, Singh et al.”
• TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, unknown): “Multiple models have been introduced recently which ablate various components of M4C for better accuracy [22, 13, 16, 20].”