TAP: Text-Aware Pre-training for Text-VQA and Text-Caption

By Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei A. F. Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, Jiebo Luo. 2020

164 citations recorded by the source corpus.

In this paper, we propose Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks. These two tasks aim at reading and understanding scene text in images for question answering and image caption generation, respectively. In contrast to conventional vision-language pretraining that fails to capture scene text and its relationship with the visual and text modalities, TAP explicitly incorporates scene text (generated from OCR engines) during pretraining. With three pre-training tasks, including masked language modeling (MLM), image-text (contrastive) matching (ITM), and relative (spatial) position prediction (RPP), pre-training with scene text effectively helps the model learn a better aligned representation among the three modalities: text word, visual object, and scene text. Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5:4%, compared with a non-TAP baseline. To further improve the performance, we build a large-scale scene text-related imagetext dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1:4 million images with scene text. Pre-trained on this OCR-CC dataset, our approach outperforms the state of the art by large margins on multiple tasks, i.e., +8:3% accuracy on TextVQA, +8:6% accuracy on ST-VQA, and +10:2 CIDEr score on TextCaps.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Since the birth of ViLBERT [30] and LXMERT [40], we have witnessed a boom of methods for vision-language pre-training [6,7,13,22,26, 37, 45, 46].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “For the scene-text related VQA tasks, existing approaches (Yang et al., 2021c; Hu et al., 2020) typically leverages the OCR engine to generate the scene text and use dynamic pointer network to decide the current output token should be OCR or the general text.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “We fine-tune using OCR strings generated by publicly available automatic service, similar to the protocol in TAP (Yang et al., 2021) and Mia (Qiao et al., 2021).”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, background): “8 million unlabeled English word images from CC-OCR [59] for visual pre-training.”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “We experiment with the following variants of decoder embedding [14,34].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Another dataset designed for pretraining for this variant isOCR-CC [202], which is a subset of images containing meaningful text taken from the CC3M dataset [130] and automatically annotated through a commercial OCR system.”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Classic VL benchmarks focus on specific capabilities of interest, such as visual recognition [31], image description [13, 1], as well as other benchmarks for specialized capabilities such as scene text understanding [65, 64, 77], commonsense reasoning [79], outside knowledge [53].”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “, 2020) and objectives (Yang et al., 2021) have been proposed for understanding natural images containing short segments of text (e.”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, methodology): “Even when they do, the amount of text is often sparse (pre-1 vious statistics suggest a median of only 6 words per image [64, 70]).”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “…including Conceptual Captions (CC3M/CC12M) (Changpinyo et al., 2021), COCO (Lin et al., 2014), Laion (Schuhmann et al., 2022), COYO (Byeon et al., 2022), DataComp (Gadre et al., 2023), Wukong (Gu et al., 2022), ImageNet (Deng et al., 2009), OCR-CC (Yang et al., 2021) and SBU (Ordonez et al., 2011).”
Playing Lottery Tickets with Vision and Language (2021, unknown): “Rewinding does not have a notable effect on the VQA performance, with only minor performance improvement observed at high-sparsity ratio (90%).”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Since then, we have witnessed a boom of VLP methods [36, 33, 63, 79, 24, 77, 9, 38].”
InfographicVQA (2021, methodology): “For VQA on images with scene text, M4C and TAP [55] use a multimodal transformer block to fuse embeddings of question, scene text tokens, and objects detected from an image.”
Structured Multimodal Attentions for TextVQA (2020, methodology): “[11] are aming at proposing pre-training tasks to learn a better aligned representation of of text word, visual object, and scene text and make performance improvement on text based tasks, such as TextVQA and Textcaps [26].”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “[200] proposed three pre-training tasks to effectively learn better aligned representations among three kinds of input data: text word, visual object, and scene text.”
TransVG: End-to-End Visual Grounding with Transformers (2021, methodology): “Motivated by the powerful pre-trained model of BERT [12], some researchers start to investigate visual-linguistic pre-training (VLP) [10, 26, 30, 43, 57] to jointly represent images and texts.”