Meshed-Memory Transformer for Image Captioning

By Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara. 2019

1,111 citations recorded by the source corpus.

Transformer-based architectures represent the state of the art in sequence modeling tasks like machine translation and language understanding. Their applicability to multi-modal contexts like image captioning, however, is still largely under-explored. With the aim of filling this gap, we present M² - a Meshed Transformer with Memory for Image Captioning. The architecture improves both the image encoding and the language generation steps: it learns a multi-level representation of the relationships between image regions integrating learned a priori knowledge, and uses a mesh-like connectivity at decoding stage to exploit low- and high-level features. Experimentally, we investigate the performance of the M² Transformer and different fully-attentive models in comparison with recurrent ones. When tested on COCO, our proposal achieves a new state of the art in single-model and ensemble configurations on the "Karpathy" test split and on the online test server. We also assess its performances when describing objects unseen in the training set. Trained models and code for reproducing the experiments are publicly available at: https://github.com/aimagelab/meshed-memory-transformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “• Caption-Transformer: We use a CNN-Transformer-based captioning model (Cornia et al., 2020) for caption-based pretraining, which recently achieves state-of-the-art results on the COCO image captioning benchmark (Lin et al.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, neutral): “Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We train our model and all the baselines in reinforcement learning setting following the work in [13].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “State-of-the-art V+L models heavily rely on self-attention mechanisms [73] or similar com-5 ponents that allow free information ﬂow among visual regions, and image captioning is no exception [66, 80, 19, 32, 23].”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “Recently, transformer-based methods [41] have been applied to image captioning [15, 12, 7], which eliminate the LSTM in the decoder.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “For example, Song et al. (2019); Wang, Chen, and Hu (2019); Gao et al. (2019); Huang et al. (2019); Pan et al. (2020); Guo et al. (2020); Cornia et al. (2020) explore different attention mechanisms in captioning modeling.”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “For example, in [94], a Transformer model is implemented for image cap-15 tioning.”
Text Augmentation Using BERT for Image Captioning (2020, methodology): “Additionally, we have confirmed our results on the other state-of-the-art models, such as [41,42], on the best variant of dataset chosen based on the previous experiments with [40].”
Unifying Multimodal Transformer for Bi-directional Image and Text Generation (2021, methodology): “The dominant image captioning models adopt encoder-decoder architectures, which embed images with a CNN encoder, and generate captions with an RNN [2, 40] or Transformer decoder [7, 45].”
Transformer Network for Remaining Useful Life Prediction of Lithium-Ion Batteries (2022, positive): “The Multi-Head Self-Attention sub-layer aims to capture the dependencies between features and ignores their distances in the sequence [54]–[57].”
Cross-modal Memory Networks for Radiology Report Generation (2022, background): “Among these studies, the most related study from Cornia et al. (2019) also proposed to leverage memory matrices to learn a priori knowledge for visual features using memory networks (Weston et al., 2015; Sukhbaatar et al., 2015; Zeng et al., 2018; Santoro et al., 2018; Nie et al., 2020; Diao et…”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, background): “, ‘the,’ ‘and’) [45], generating fine, novel and diverse sentences [7,24,70], and incorporating information from different semantic taggers [12, 33].”
AdaBins: Depth Estimation Using Adaptive Bins (2020, background): “Third, several previous architectures propose performing global processing using attention blocks to process information after an encoder block in the architecture (e.g., image captioning [5, 18] or object detection [2]).”
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “…and visual information; Herdade et al. (2019) introduced an Object Relation Transformer that models the spatial relationship between objects; Cornia et al. (2020) presented Meshed-Memory Transformer ( M 2 Trans) that learns relationships between input regions and exploits hierarchical…”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “These methods are slower to train but are generally smaller than Transformer-based ones (apart from the optimized M 2 Transformer model [69]).”
A Survey of Transformers (2021, background): “, visual question answering [55, 75, 77, 124], visual commonsense reasoning [75, 124], caption generation [22, 81, 127], speech-to-text translation [46] and text-to-image generation [29, 81, 107].”
Dual-Level Collaborative Transformer for Image Captioning (2021, methodology): “To explain, most recent methods in image captioning (Huang et al. 2019; Cornia et al. 2020; Pan et al. 2020) use the self-attention modules to model the relationships of visual features.”
Generating Radiology Reports via Memory-driven Transformer (2020, background): “Generating radiology reports is essentially an image-to-text generation task, for which there exist several solutions (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018; Cornia et al., 2019).”