OCR-VQA: Visual Question Answering by Reading Text in Images
By Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, Anirban Chakraborty. 2019
681 citations recorded by the source corpus.
The problem of answering questions about an image is popularly known as visual question answering (or VQA in short). It is a well-established problem in computer vision. However, none of the VQA methods currently utilize the text often present in the image. These "texts in images" provide additional useful cues and facilitate better understanding of the visual content. In this paper, we introduce a novel task of visual question answering by reading text in images, i.e., by optical character recognition or OCR. We refer to this problem as OCR-VQA. To facilitate a systematic way of studying this new problem, we introduce a large-scale dataset, namely OCRVQA-200K. This dataset comprises of 207,572 images of book covers and contains more than 1 million question-answer pairs about these images. We judiciously combine well-established techniques from OCR and VQA domains to present a novel baseline for OCR-VQA-200K. The experimental results and rigorous analysis demonstrate various challenges present in this dataset leaving ample scope for the future research. We are optimistic that this new task along with compiled dataset will open-up many exciting research avenues both for the document image analysis and the VQA communities.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Cordelia Schmid
- Andrew Zisserman
In Multimodal Learning
In Question Answering & Reading Comprehension
- Anton van den Hengel
- Qi Wu
- Devi Parikh
- Dhruv Batra
- Jiasen Lu
- Chunhua Shen
- Radu Soricut
- Alexander Gerhard Schwing
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “∗ Training images of GQA were observed during training. four additional datasets that are used in InstructBLIP: open-knowledge VQA (OKVQA [40], A-OKVQA [44]) and OCR (OCRVQA [41], TextCaps [46]).”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “As illustrated in Table 4, we achieve new SOTA on VizWiz-VQA and OCR-VQA, and same performance with prior SOTA of LaTr (Biten et al., 2022) on ST-VQA.”
- TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “Meanwhile, in Visual Question Answering, multiple datasets [6,26,33] were recently introduced which focus on text-based visual question answering.”
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, methodology): “We assess our models’ ability toward text-oriented visual question answering on several benchmarks including TextVQA (Sidorov et al., 2020), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), AI2Diagram (Kembhavi et al., 2016), and OCR-VQA (Mishra et al., 2019).”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “For OCR tasks, we utilized large-scale datasets such as Wukong-OCR, LaionCOCO-OCR, and Common Crawl PDFs, which constitute 32.0% of our data.”
- The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “The model is capable of reading some simple signs, but, as highlighted by [37], reasoning about the semantics of written text placed in images remains a challenge, e.”
- Spatially Aware Multimodal Transformers for TextVQA (2020, methodology): “Specifically, early models [6,7,32] proposed for the TextVQA task did not encode any explicit spatial information while M4C [14] simply adds a location embedding of the absolute location to the input feature.”
- ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “We employ 9 benchmarks in the VQA task, namely DocVQA [73], TextVQA [74], STVQA [75], OCR-VQA [76], OKVQA [77], GQA [78], IconQA [79], Visual Spatial Reasoning (VSR) [80], and Visual Dialog (Visdial) [81].”
- DoRA: Weight-Decomposed Low-Rank Adaptation (2024, methodology): “The fine-tuned models are then evaluated on seven vision-language benchmarks: VQA v2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), VisWiz (Gurari et al., 2018) SQA (Lu et al., 2022), VQA T (Singh et al., 2019), POPE (Li et al., 2023), and MMBench (Liu et al., 2023c).”
- VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, methodology): “For VQA benchmarks [35, 39], we follow the standard practices to calculate accuracies based on heuristic matching.”
- MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “For VQA datasets, our training takes a variety of datasets, such as GQA [19], VQA-v2 [14], OCR-VQA [31], OK-VQA [30], and AOK-VQA [43].”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…SynthDoG-RU [112], SynthDoG-JP [112], SynthDoG-KO [112], IAM [180], EST-VQA [253], ST-VQA [17], NAF [52], InfoVQA [183], HME100K [288], OCRVQA [188], SROIE [97], POIE [115], CTW [287], SynthText [79], ArT [40], LSVT [222], RCTW-17 [209], ReCTs [301], MTWI [82], TextVQA [212], CASIA [146],…”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Benchmark datasets The Image Captioning and VQA benchmarks used for evaluation is summarized in Appendix B, including 6 Image Captioning benchmarks (COCO (Karpathy split [30]), NoCaps [31], TextCaps [32], VizWiz-Cap [33], Screen2Words [34], Widget-Cap [35]) and 13 VQA benchmarks (VQAv2 [36], OKVQA [”
- LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (2023, methodology): “InstructBLIP [31] transforms 13 vision-language tasks (including OCR-VQA [32]) into the instruction-following format for instruction tuning.”
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “…script, we utilize the developed LLMs like GPT-4 [44] and Claude-2 [3] to generate movie summaries, plot-related QA pairs, and general reasoning QA pairs. tions from ShareGPT [1], 625K single-or multi-turn visual QA pairs from [19, 22, 25, 26, 33, 40–42, 47, 49], and 98K video QA pairs from [7].”