Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA

By Ronghang Hu, Amanpreet Singh, Trevor Darrell, Marcus Rohrbach. 2019

235 citations recorded by the source corpus.

Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, we propose a novel model for the TextVQA task based on a multimodal transformer architecture accompanied by a rich representation for text in images. Our model naturally fuses different modalities homogeneously by embedding them into a common semantic space where self-attention is applied to model inter- and intra- modality context. Furthermore, it enables iterative answer decoding with a dynamic pointer network, allowing the model to form an answer through multi-step prediction instead of one-step classification. Our model outperforms existing approaches on three benchmark datasets for the TextVQA task by a large margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “For the scene-text related VQA tasks, existing approaches (Yang et al., 2021c; Hu et al., 2020) typically leverages the OCR engine to generate the”
Measuring Social Biases in Grounded Vision and Language Embeddings (2020, methodology): “Grounded embeddings are used for many conse-quential tasks in natural language processing, like visual dialog (Murahari et al., 2019) and visual question answering (Hu et al., 2019).”
Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning (2020, methodology): “The TextVQA task, with themeaning of visual question answering based on text in images, has been newly developed in recent years [5, 6, 13, 24, 31].”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[31, 38, 39, 57, 9, 30, 55, 71, 23]).”
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020, methodology): “We compare our result on TextVQA to the newest SOTA method SMA (Gao et al., 2020) and other existing works like LoRRA (Singh et al., 2019), MSFT VTI (MSFTVTI, 2019), and M4C (Hu et al., 2020).”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, methodology): “In the image captioning area, the texts contained in images are also of critical importance and often provide valuable information [5, 19, 20, 34, 41] for caption generation.”
Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[30, 37, 38, 56, 9, 29, 55, 70, 22]).”
A Survey of Transformers (2021, background): “, visual question answering [55, 75, 77, 124], visual commonsense reasoning [75, 124], caption generation [22, 81, 127], speech-to-text translation [46] and text-to-image generation [29, 81, 107].”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “We experiment with the following variants of decoder embedding [14,34].”
Spatially Aware Multimodal Transformers for TextVQA (2020, methodology): “For instance, LoRRA [38] does not use any location information while M4C [14] merely encodes the absolute location of objects and text as input to the model.”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, methodology): “Visual Features Most previous methods utilized an ex-ternal pre-trained object detector [20,70] for extracting objects labels, visual object features and visual OCR features.”
InfographicVQA (2021, methodology): “We adapt a multimodal Transformer [48]-based VQA model called M4C [21] and a layout-aware, BERT [14]- style extractive QA model called LayoutLM [52] for VQA on InfographicVQA.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, methodology): “, M4C [19]) on three benchmarks in Sec.”