Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning

By Jing Wang, Jinhui Tang, Jiebo Luo. 2020

57 citations recorded by the source corpus.

OCR-based image captioning is the task of automatically describing images based on reading and understanding written text contained in images. Compared to conventional image captioning, this task is more challenging, especially when the image contains multiple text tokens and visual objects. The difficulties originate from how to make full use of the knowledge contained in the textual entities to facilitate sentence generation and how to predict a text token based on the limited information provided by the image. Such problems are not yet fully investigated in existing research. In this paper, we present a novel design - Multimodal Attention Captioner with OCR Spatial Relationship (dubbed as MMA-SR) architecture, which manages information from different modalities with a multimodal attention network and explores spatial relationships between text tokens for OCR-based image captioning. Specifically, the representations of text tokens and objects are fed into a three-layer LSTM captioner. Different attention scores for text tokens and objects are exploited through the multimodal attention network. Based on the attended features and the LSTM states, words are selected from the common vocabulary or from the image text by incorporating the learned spatial relationships between text tokens. Extensive experiments conducted on the TextCaps dataset verify the effectiveness of the proposed MMA-SR method. More remarkably, our MMA-SR increases CIDEr-D score from 93.7% to 98.0%.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jianfeng Gao
3. Dacheng Tao
4. Lijuan Wang
5. Cordelia Schmid
6. Trevor Darrell
7. Xiaogang Wang
8. Andrew Zisserman

In Multimodal Learning

1. Jianfeng Wang
2. Lijuan Wang
3. Zhengyuan Yang
4. Jiebo Luo
5. Qi Wu
6. Zhe Gan
7. Lei Zhang
8. Hexiang Hu

Citations in the corpus

• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “Similar explorations [55] on the spatial relationships are studied in the Text-Caption task.”
• Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “Additionally, a variety of model architectures (Singh et al., 2019; Sidorov et al., 2020; Wang et al., 2020) and objectives (Yang et al., 2021) have been proposed for understanding natural images containing short segments of text (e.g. street signs).”
• Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, methodology): “Most image captioning models [2, 15, 42, 44, 45, 49] use CNNs to encode visual features and apply RNNs as language decoder to generate descriptions.”