Areas of Attention for Image Captioning

By Marco Pedersoli, Thomas Lucas, Cordelia Schmid, Jakob Verbeek. 2016

218 citations recorded by the source corpus.

We propose “Areas of Attention”, a novel attentionbased model for automatic image captioning. Our approach models the dependencies between image regions, caption words, and the state of an RNN language model, using three pairwise interactions. In contrast to previous attentionbased approaches that associate image regions only to the RNN state, our method allows a direct association between caption words and image regions. During training these associations are inferred from image-level captions, akin to weakly-supervised object detector training. These associations help to improve captioning by localizing the corresponding regions during testing. We also propose and compare different ways of generating attention areas: CNN activation grids, object proposals, and spatial transformers nets applied in a convolutional fashion. Spatial transformers give the best results. They allow for image specific attention areas, and can be trained jointly with the rest of the network. Our attention mechanism and spatial transformer attention areas together yield state-of-the-art results on the MSCOCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “Recently, integrating image regions eventually extracted from a detector as attention candidates has become the predominant strategy in captioning architectures [25], [8].”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, background): “In this paper we proposed a technique for continual learning of image captioning networks based on Recurrent Attention to Transient Tasks (RATT).”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “On the image encoding side, instead, single-layer attention mechanisms have been adopted to incorporate spatial knowledge, initially from a grid of CNN features [45, 26, 50], and then using image regions extracted with an object detector [4, 29, 27].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “[112] proposed an area-based attention mechanism for image captioning.”
Multi-Scale Self-Guided Attention for Medical Image Segmentation (2019, background): “Many extensions of these networks have been proposed to solve pixel-wise segmentation problems in a wide range of applications [30]–[39].”
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (2018, background): “Generally, all integrate recurrent neural networks as language models, and a representation of the image which might be given by the output of one or more layer of a CNN [43, 10, 37, 24], or by a time-varying vector extracted with an attention mechanism [46, 48, 24, 7, 3] selected either from a grid”
A Corpus of Natural Language for Visual Reasoning (2017, methodology): “Our task is related to caption generation, which has been studied extensively (e.g., Pedersoli et al., 2016; Carrara et al., 2016; Chen et al., 2016) with MSCOCO (Chen et al., 2015b) and Flickr30K (Young et al., 2014; Plummer et al., 2015).”
Bottom-Up and Top-Down Attention for Image Captioning and VQA (2017, methodology): “[33] use spatial transformer networks [19] to generate image features, which are processed using a custom attention model based on three bi-linear pairwise interactions.”
Generalizing Face Forgery Detection with High-frequency Features (2021, background): “The attention mechanism has been broadly applied in natural language processing [48] and computer vision [22, 37, 23].”
Resources and End-to-End Neural Network Models for Arabic Image Captioning (2020, unknown): “…mechanism was adopted to focus on salient regions of the image while generating words (Xu et al., 2015) (Wu and Cohen, 2016) (Jin et al., 2015) (Pedersoli et al., 2016) (Lu et al., 2017) (Liu et al., 2017) (Chen et al., 2017) (Tavakoliy et al., 2017) (Anderson et al., 2017) (Chunseong Park et…”
Fooling Vision and Language Models Despite Localization and Attention Mechanism (2017, background): “Recently [61] proposed an end-to-end model which regresses a set of image regions and learns to associate caption words to these regions.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “A modular approach is also explored where three modular components related to subject appearance, location, and relationship to other objects is used to model with Modular Attention Network (Yu et al., 2018).”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “The Areas of Attention captioning model [30] uses either edge boxes [50] or spatial transformer networks [17] to generate image features, which are processed using an attention model based on three bi-linear pairwise interactions [30].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Many subsequent works have adopted a decoder based on a single-layer LSTM, mostly without any architectural changes [50], [51], [67], while others have proposed significant modifications, summarized below.”
Cross Attention Network for Few-shot Classification (2019, background): “It has achieved great success in computer vision applications, such as image classification [12, 41, 27], person re-identification [10, 47, 11], image caption [29, 44, 4] and visual question answering [43, 45, 46, 30].”
Entangled Transformer for Image Captioning (2019, background): “On the one hand, visual attention exploits the low-level feature maps [41] or high-level object ROI-pooled features [29, 2] to identify the most relevant regions for the words.”
TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays (2018, background): “[29] allowed a direct association between caption words and image regions.”