Scene Text Visual Question Answering

By Ali Furkan Biten, Rubèn Pérez Tito, Andrés Mafla, Lluís Gómez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, Dimosthenis Karatzas. 2019

512 citations recorded by the source corpus.

Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the Visual Question Answering process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “, the TextVQA dataset [49], the ST-VQA dataset [8], etc.”
Achieving Human Parity on Visual Question Answering (2021, methodology): “For training Text Reading Expert, three text-reading VQA datasets are used, including a subset of VQA data [4], TextVQA [71], and ST-VQA [5].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “As illustrated in Table 4, we achieve new SOTA on VizWiz-VQA and OCR-VQA, and same performance with prior SOTA of LaTr (Biten et al., 2022) on ST-VQA.”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “To situate TextCaps properly w.r.t. other image captioning datasets, we compare TextCaps with other prominent image captioning datasets, namely COCO [9], SBU [26], and Conceptual Captions [29], as well as reading-oriented VQA datasets TextVQA [31], ST-VQA [6], and OCR-VQA [25].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “, 2018) and ST-VQA (Biten et al., 2019) require the ability to perform question answering in the presence of text in the input image.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “…[20], TextCaps [96] VQAv2 [40], OKVQA [79], A-OKVQA [93], VQA 1.1M IconQA [75], AI2D [53], GQA [48] OCR-VQA [82], ChartQA [80], DocVQA [25], ST-VQA [11], EST-VQA [116], InfoVQA [81] model can generate a feature map F ∈ R H/ 14 × W/ 14 × D for dense prediction tasks, or work with global average…”
What matters when building vision-language models? (2024, unknown): “…39,463 337,829 3.46% TextCaps (Sidorov et al., 2020) 21,953 21,953 389,658 2.00% TextVQA (Singh et al., 2019) 21,953 34,602 181,918 1.86% ST-VQA (Biten et al., 2019) 17,247 23,121 127,846 1.31% OCR-VQA (Mishra et al., 2019) 165,746 801,579 6,073,824 0.93% VisualMRC (Tanaka et al., 2021) 3,027…”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, background): “In the image captioning area, the texts contained in images are also of critical importance and often provide valuable information [5, 19, 20, 34, 41] for caption generation.”
On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering (2020, methodology): “A variety of text-based VQA datasets [4, 15, 24, 29] have been proposed.”
RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition (2020, background): “lidating its robustness in both contextual and contextless application scenarios. 1 Introduction Scene text recognition is crucial for visual understanding and reasoning in many application scenarios [44,5]. Despite great progress recently, it remains a challenging task because the potential irregul”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “Finally, we conduct experiments with our proposed MM-GNN and its variants on two recently proposed datasets TextVQA [44] and ST-VQA [8].”
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “…from 165k images by prompting 21], Infograph-icVQA train [22], AI2D train (transparent and opaque label boxes) [13], A-OKVQA train [28], Android-Control train [14], ScienceQA train [16], TabMWP train [17], ST-VQA train [6], TallyQA train [3], DVQA train [11], FigureQA train [12], and train [23].”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, unknown): “The responses generated by LMM often include many explanatory terms, so the exact matching approach or Average Normalized Levenshtein Similarity(ANLS) [36] used in the original dataset are not suitable for evaluating LMM in zero-shot scenarios.”
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020, methodology): “We evaluate our LaAP-Net on the three challenging benchmark datasets: TextVQA (Singh et al., 2019), ST-VQA (Biten et al., 2019a) and OCR-VQA (Mishra et al., 2019).”
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA (2019, methodology): “Recently, a few datasets and methods [44, 8, 37, 7] have been proposed for visual question answering based on text in images (referred to as the TextVQA task).”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “…[96], ArT (en&zh) [21], COCO-Text (en) [109], CTW (zh) [129], LSVT (zh) [100], RCTW-17 (zh) [94], ReCTs (zh) [132], OCR SynthDoG (en&zh) [41], ST-VQA (en) [11] Document DocVQA (en) [22], Common Crawl PDF (en&zh) Grounding RefCOCO/+/g (en) [75, 126], Visual Genome (en) [42] LLaVA-150K (en&zh) [59],…”
InfographicVQA (2021, methodology): “TQA [24] Science diagrams ✗ ✗ MR 1K 26K MCQ RecipeQA [48] Culinary pictures ✗ ✓ MR 251K 37K MCQ ST-VQA [7] Natural images ✗ ✗ ST 23K 31K Ex TextVQA [39] Natural images ✗ ✗ ST 28K 45K Ex, SAb OCR-VQA [32] Book covers ✗ ✓ BD 207K 1M Ex, Y/N DVQA [21] Bar charts ✓ ✓ BD 300K 3.”