OCR-VQA: Visual Question Answering by Reading Text in Images

By Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, Anirban Chakraborty. 2019

681 citations recorded by the source corpus.

The problem of answering questions about an image is popularly known as visual question answering (or VQA in short). It is a well-established problem in computer vision. However, none of the VQA methods currently utilize the text often present in the image. These "texts in images" provide additional useful cues and facilitate better understanding of the visual content. In this paper, we introduce a novel task of visual question answering by reading text in images, i.e., by optical character recognition or OCR. We refer to this problem as OCR-VQA. To facilitate a systematic way of studying this new problem, we introduce a large-scale dataset, namely OCRVQA-200K. This dataset comprises of 207,572 images of book covers and contains more than 1 million question-answer pairs about these images. We judiciously combine well-established techniques from OCR and VQA domains to present a novel baseline for OCR-VQA-200K. The experimental results and rigorous analysis demonstrate various challenges present in this dataset leaving ample scope for the future research. We are optimistic that this new task along with compiled dataset will open-up many exciting research avenues both for the document image analysis and the VQA communities.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Cordelia Schmid
  8. Andrew Zisserman

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Linjie Li
  4. Devi Parikh
  5. Zicheng Liu
  6. Jianfeng Wang
  7. Qi Wu
  8. Jianfeng Gao

In Question Answering & Reading Comprehension

  1. Anton van den Hengel
  2. Qi Wu
  3. Devi Parikh
  4. Dhruv Batra
  5. Jiasen Lu
  6. Chunhua Shen
  7. Radu Soricut
  8. Alexander Gerhard Schwing

Citations in the corpus