Spatially Aware Multimodal Transformers for TextVQA

By Yash Kant, Dhruv Batra, Peter Anderson, Alexander Gerhard Schwing, Devi Parikh, Jiasen Lu, Harsh Agrawal. 2020

97 citations recorded by the source corpus.

Textual cues are essential for everyday tasks like buying groceries and using public transport. To develop this assistive technology, we study the TextVQA task, i.e., reasoning about text in images to answer a question. Existing approaches are limited in their use of spatial relations and rely on fully-connected transformer-like architectures to implicitly learn the spatial structure of a scene. In contrast, we propose a novel spatially aware self-attention layer such that each visual entity only looks at neighboring entities defined by a spatial graph. Further, each head in our multi-head self-attention layer focuses on a different subset of relations. Our approach has two advantages: (1) each head considers local context instead of dispersing the attention amongst all visual entities; (2) we avoid learning redundant features. We show that our model improves the absolute accuracy of current state-of-the-art methods on TextVQA by 2.2% overall over an improved baseline, and 4.62% on questions that involve spatial reasoning and can be answered correctly using OCR tokens. Similarly on ST-VQA, we improve the absolute accuracy by 4.2%. We further show that spatially aware self-attention improves visual grounding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Linjie Li
3. Lijuan Wang
4. Zicheng Liu
5. Jianfeng Gao
6. Jianfeng Wang
7. Xiaowei Hu
8. Zhengyuan Yang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Cordelia Schmid
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Andrew Zisserman
7. Trevor Darrell
8. Zhe Gan

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Jianfeng Wang
5. Zicheng Liu
6. Xiaowei Hu
7. Zhengyuan Yang
8. Qi Wu

Citations in the corpus

• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “SA-M4C [25] further improved M4C by encoding the spatial relationships among visual regions as the attention masks in the multi-modal transformer.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[50] Yash Kant, Dhruv Batra, Peter Anderson, Alexander G.”
• TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021, background): “Multiple models have been introduced recently which ablate various components of M4C for better accuracy [22, 13, 16, 20].”
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “Transformers have also been used in a variety of vision image tasks, such as image representation learning [12, 82, 18, 59], image generation [51], object detection [48, 10], video question-answering [35], few-shot learning [19], and image–text [49, 63, 67, 44, 68], video-text [65, 64, 89, 25, 53, 2”
• Structured Multimodal Attentions for TextVQA (2020, background): “Following on M4C, the LaAP [21] further predicts the OCR position as the evidence for final predicted answer, SA-M4C [22] replaced the original self-attention layers with their novel spatially aware selfattention layers, Singh et al.”
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, methodology): “Inspired by [14, 20, 43], we calculate the similarity between each paired regions by their Intersection over Union (IoU) score.”
• LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, methodology): “SA-M4C [25] build on top of M4C by providing supervision on self-attention weights.”