TextCaps: a Dataset for Image Captioning with Reading Comprehension

By Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh. 2020

577 citations recorded by the source corpus.

Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “M4C’s variants M4CCaptioner [46] set a strong baseline on TextCaps [46] with the question text inputs removed.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The dual encoder approach works well for unimodal [107, 108] and cross-modal retrieval tasks [66, 81] but their lack of fusion usually causes them to underperform on tasks that involve visual reasoning and question answering [39, 53, 93, 96] which is where models based on fusion encoder(s) shine.”
Improved Baselines with Visual Instruction Tuning (2023, methodology): “∗ Training images of GQA were observed during training. four additional datasets that are used in InstructBLIP: open-knowledge VQA (OKVQA [40], A-OKVQA [44]) and OCR (OCRVQA [41], TextCaps [46]).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “, 2019)2 which focuses on novel objects, TextCaps (Sidorov et al., 2020) which focuses on scene-text understanding, and VizWiz-Captions (Gurari et al.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “2020b), image captioning with reading comprehension (Sidorov et al. 2020) have been studied.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “, 2019) and TextCaps (Sidorov et al., 2020) datasets are shown, together with PaLI inputs and outputs.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “The Prismatic VLM is fine-tuned on top of these components using the LLaVA 1.5 data mixture [43], which contains a total of approximately 1M image-text and text-only data samples from open-source datasets [29, 42, 81–83].”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, unknown): “Caption COCO [8], SUB [8], TextCaps [59] VQA VQAv2 [1], GQA [26], OK-VQA [44], VSR [35], IConQA [42] Text-VQA [60], SQA [41], OCR-VQA [45], Visual Dialog [12], LLava-150k [38] Instruction tuning Image-Text Composiiton In-house data (Refer to Sec.3.3) Conversation LLaVA-150k [38], Alpaca-en&zh [61],…”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, methodology): “When training MAPL’s mapping network on only 1% of VizWiz data, we believe the data is not large enough to compensate for the OOD pretrained vision encoder, so MAPL trained on COCO/TextCaps performs better on VizWiz-VQA.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, methodology): “Images and captions sourced from TextCaps (images with text/logos; Sidorov et al., 2020).”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “Additionally, a variety of model architectures (Singh et al., 2019; Sidorov et al., 2020; Wang et al., 2020) and objectives (Yang et al., 2021) have been proposed for understanding natural images containing short segments of text (e.g. street signs).”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “The task was introduced by Sidorov et al. [137] with the TextCaps dataset.”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “We also integrate Paragraph Captioning [33], TextCaps [66], and TextVR [83], which require uniquely comprehending text within images and videos.”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Classic VL benchmarks focus on specific capabilities of interest, such as visual recognition [31], image description [13, 1], as well as other benchmarks for specialized capabilities such as scene text understanding [65, 64, 77], commonsense reasoning [79], outside knowledge [53].”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, methodology): “We assess our models’ ability toward text-oriented visual question answering on several benchmarks includingTextVQA(Sidorov et al., 2020), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), AI2Diagram (Kembhavi et al., 2016), and OCR-VQA (Mishra et al., 2019).”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “∈ R d × d learnable projection matrices, and C l ∈ R ( L V + L T ) × d context features of l -th layer.”
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “Its corresponding Parital LoRA contains two low-rank matrix we have the output feature ˆ x by: ShareGPT4V-PT [11], COCO [13], Nocaps [1], TextCaps [73], LAION400M [69], SBU [59], CC”
DoRA: Weight-Decomposed Low-Rank Adaptation (2024, background): “…contain several datasets from VQA (Goyal et al., 2017; Hudson & Manning, 2019; Marino et al., 2019; Schwenk et al., 2022), OCR (Mishra et al., 2019; Sidorov et al., 2020), region-level VQA (Kazemzadeh et al., 2014; Krishna et al., 2017; Mao et al., 2016), visual conversation (Liu et al., 2023a),…”