LaTr: Layout-Aware Transformer for Scene-Text VQA

By Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, Raghavan Manmatha. 2021

121 citations recorded by the source corpus.

We propose a novel multimodal architecture for Scene Text Visual Question Answering (STVQA), named Layout-Aware Transformer (LaTr). The task of STVQA requires models to reason over different modalities. Thus, we first investigate the impact of each modality, and reveal the importance of the language module, especially when enriched with layout information. Accounting for this, we propose a single objective pre-training scheme that requires only text and spatial cues. We show that applying this pre-training scheme on scanned documents has certain advantages over using natural images, despite the domain gap. Scanned documents are easy to procure, text-dense and have a variety of layouts, helping the model learn various spatial cues (e.g. left-of, below etc.) by tying together language and layout information. Compared to existing approaches, our method performs vocabulary-free decoding and, as shown, generalizes well beyond the training vocabulary. We further demonstrate that LaTr improves robustness towards OCR errors, a common reason for failure cases in STVQA. In addition, by leveraging a vision transformer, we eliminate the need for an external object detector. LaTr outperforms state-of-the-art STVQA methods on multiple datasets. In particular, +7.6% on TextVQA, +10.8% on ST-VQA and +4.0% on OCR-VQA (all absolute accuracy numbers).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “, 2021c),ST-VQA (Biten et al., 2022),VizWiz-VQA (Alayrac et al.”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, methodology): “Some components of such models, e.g. T5 and TaPas (Eisenschlos et al., 2020) used in the VL-T5 and VisionTaPas models of Masry et al. (2022) or LATr’s OCR output encoder (Biten et al., 2022) have been pretrained on digital-born or OCR-ed documents.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, unknown): “5 - - - [5] [5] [52] [5] [5] [53] [54] [54] [45] [46] - - - PaLI-X 163.”