Multimodal Transformer With Multi-View Visual Representation for Image Captioning

By Jun Yu, Jing Li, Zhou Yu, Qingming Huang. 2019

450 citations recorded by the source corpus.

Image captioning aims to automatically generate a natural language description of a given image, and most state-of-the-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN) based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intra-modal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches, the MT model simultaneously captures intra- and inter-modal interactions in a unified attention block. Due to the in-depth modular composition of such attention blocks, the MT model can perform complex multimodal reasoning and output accurate captions. Moreover, to further improve the image captioning performance, multi-view visual features are seamlessly introduced into the MT model. We quantitatively and qualitatively evaluate our approach using the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method significantly outperforms the previous state-of-the-art methods. With an ensemble of seven models, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge at the time of the writing of this paper.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “…of bounding region proposals as our methods (ﬁrst section), our models (second section), and other incomparable methods (third section) that use external question-answer pairs from Visual Genome (+VG) , multiple detectors (Yu et al., 2019a) (+Multiple Detectors) and ensembles of their models.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “State-of-the-art V+L models heavily rely on self-attention mechanisms [73] or similar com-5 ponents that allow free information ﬂow among visual regions, and image captioning is no exception [66, 80, 19, 32, 23].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
Connecting Vision and Language with Localized Narratives (2019, methodology): “We start from a state-of-the-art, transformer-based encoder-decoder image captioning model [49, 7, 62].”
Deep Multimodal Data Fusion (2024, background): “There are various other multimodal tasks, e.g., visual question answering ( VQA ) (text-image: [154, 220], text-video: [82, 223]), RGB-depth object segmentation [31, 39], medical data analysis [181, 185], and image captioning [216, 237].”
Modeling question asking using neural program generation (2019, methodology): “Our model uses a convolutional encoder to represent the game state, and a Transformer de-coder for generating questions in a domain speciﬁc language (DSL).”
An Edge Traffic Flow Detection Scheme Based on Deep Learning in an Intelligent Transportation System (2021, methodology): “Currently, the popular algorithms of object detection and multiple-object tracking, e.g., autonomous platooning, based on deep learning adopt the deep convolution neural network [27].”