Image Captioning: Transforming Objects into Words

By Simão Herdade, Armin Kappeler, Kofi Boakye, Joao Soares. 2019

581 citations recorded by the source corpus.

Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Scaling Up Vision-Language Pretraining for Image Captioning (2021, neutral): “Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57].”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works have used LSTM variants [8, 38, 39], while recent works [16, 26] adopted the improved transformer architecture [36].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “In particular, we start from our model without memory, and keep the 1 https://spacy.io/ Noticeably, the performance obtained with six layers (as in the original Transformer model [10], and as in the captioning model of [11]) is lower than the one obtained when using 1, 2 or 3 layers.”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “Recently, transformer-based methods [41] have been applied to image captioning [15, 12, 7], which eliminate the LSTM in the decoder.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “Finally, we compare with ORT [13], which uses a plain Transformer and weights attention scores in the region encoder with pairwise distances between detections.”
Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, background): “Image captioning [3, 20, 17, 7, 51] aims to generate a natural language sentence to describe the image content.”
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “Li et al. (2019) introduced an entangled attention that enables Transformer to jointly exploit semantic and visual information; Herdade et al. (2019) introduced an Object Relation Transformer that models the spatial relationship between objects; Cornia et al. (2020) presented Meshed-Memory…”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
Translation between Molecules and Natural Language (2022, background): “Image captioning, for example, has made significant progress (Herdade et al., 2019; Pan et al., 2020), and recent datasets for the task have also grown to large sizes (Chen et al.”
Dual-Level Collaborative Transformer for Image Captioning (2021, background): “GCN-LSTM (Yao et al. 2018) and Object Relation Transformer (Herdade et al. 2019) utilize bounding boxes of regions to model location relationships between regions in a relative manner.”
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “…i , cy i ) , relative width w i , and relative height h i of grid i as follows: Finally, we imitate the computation of region geometry features in [12, 9] to obtain the relative geometry features be-tween two grids i and j : where r ∈ R N × N × 4 is the relative geometry relationship between…”
Syntax-Aware Action Targeting for Video Captioning (2020, background): “Compared with image captioning [2, 31, 20, 8] that aims at depicting the static scene in an image, video captioning emphasizes more on the action and attracts increasing attention in the ﬁeld of both computer vision and artiﬁcial intelligence.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Geometry-aware encoding – Herdade et al. [77] introduced a modified version of self-attention that takes into account the spatial relationships between regions.”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “Other works proposed variants or modifications of the self-attention operator tailored for image captioning [65], [66], [67], [68], [69].”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “On the text modeling side of the problem, language models (LMs) have also advanced with the development of LSTMs [16, 69], CNNs [4] and Transformers [27, 29, 47].”
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting (2022, background): “To address these limitations, a potential research direction is the development of automatic time series prompting or time series captioning (similar to image captioning [29]), which utilizes generative models to describe time series data.”