Towards VQA Models That Can Read

By Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, Marcus Rohrbach. 2019

2,180 citations recorded by the source corpus.

Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today’s VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new “TextVQA” dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA). We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “To compensate, models are augmented with the outputs of custom OCR engines and features to boost performance on tasks where this capability is required (Singh et al., 2019; Yang et al., 2020).”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, neutral): “3 (Singh et al., 2019) 68.”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “Among them, LoRRA [49] added an OCR attention branch for scene text encoding to a VQA model [24].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “A Faster-RCNN model is used to extract features from o RoIs, denoted by { r 0 , · · · , r o − 1 } , from the image to represent its visual content.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “SOTA model SimVLM [55] PICa [58] CFR [33] Pythia [46] SimVLM SimVLM VIVO [22] SimVLM OSCAR [29]”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The dual encoder approach works well for unimodal [107, 108] and cross-modal retrieval tasks [66, 81] but their lack of fusion usually causes them to underperform on tasks that involve visual reasoning and question answering [39, 53, 93, 96] which is where models based on fusion encoder(s) shine.”
Achieving Human Parity on Visual Question Answering (2021, methodology): “For training Text Reading Expert, three text-reading VQA datasets are used, including a subset of VQA data [4], TextVQA [71], and ST-VQA [5].”
Improved Baselines with Visual Instruction Tuning (2023, neutral): “TextVQA [47] contains text-rich visual question answering.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Compared with the concurrent work of Flamingo (Alayrac et al., 2022), we achieve higher accuracy (+5.4) on TextVQA and lower (-3.29) on VQAv2.”
In Defense of Grid Features for Visual Question Answering (2020, methodology): “In the Pythia implementation [16], of bottom-up attention [2], a Feature Pyramid Network (FPN) model [24] is used to compute region features.”
What Does BERT with Vision Look At? (2020, neutral): “3 (Singh et al., 2019) and report on test-dev; on VCR, we compare to R2C (Zellers et al.”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “Meanwhile, in Visual Question Answering, multiple datasets [6,26,33] were recently introduced which focus on text-based visual question answering.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “TextVQA (Singh et al., 2019), VizWiz-QA (Gurari et al., 2018) and ST-VQA (Biten et al., 2019) require the ability to perform question answering in the presence of text in the input image.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…and charts (Methani et al., 2020; Masry et al., 2022; Kahou et al., 2017; Chang et al., 2022; Kafle et al., 2018), documents (text in images) (Singh et al., 2019; Mathew et al., 2022; Liu et al., 2023c), or synthetic images (Dahlgren Lindström & Abraham, 2022; Li et al., 2023; Bitton-Guetta…”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “In the table, LoRRA (Pythia) is the baseline provided by TextVQA dataset [44].”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, background): “…models on a series of multimodal benchmarks spanning visual question answering (VQAv2 (Antol et al., 2015), OKVQA (Marino et al., 2019), TextVQA (Singh et al., 2019)), hateful speech detection (HatefulMeme (Kiela et al., 2020)), image captioning (COCO (Lin et al., 2014), Flickr30k (Young et…”