Attention on Attention for Image Captioning

By Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei. 2019

1,012 citations recorded by the source corpus.

Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, neutral): “8 AoANet [22] 42.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, neutral): “2 AoANet [11] 37.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, neutral): “Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57].”
Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “We examine the captions generated by six image captioning models: (1) FC [59] is a simple sequence encoder that takes in image features encoded by a CNN; (2) Att2in [59] is similar but images are encoded using spatial features; (3) DiscCap [50] further adds a loss term to encourage discriminability;”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “• AoA Transformer, which inserts an attention-on-attention (AoA) module [24] into every transformer layer, as depicted by Figure 3 (b).”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “State-of-the-art V+L models heavily rely on self-attention mechanisms [73] or similar com-5 ponents that allow free information ﬂow among visual regions, and image captioning is no exception [66, 80, 19, 32, 23].”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “XGPT uses AoA module (Huang et al., 2019) as visual encoder.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[42] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “Recently, transformer-based methods [41] have been applied to image captioning [15, 12, 7], which eliminate the LSTM in the decoder.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “For example, Song et al. (2019); Wang, Chen, and Hu (2019); Gao et al. (2019); Huang et al. (2019); Pan et al. (2020); Guo et al. (2020); Cornia et al. (2020) explore different attention mechanisms in captioning modeling.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “Noticeably, when using four models our approach achieves the best performance according to all metrics, with an increase of 2.5 CIDEr points with respect to the current state of the art [14].”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “9 3 AoANet [18] COCO 18.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “As vision and language usually contain complementary information, joint learning of vision and language representations has been proven to be quite effective on multimodality tasks, such as Visual Question Answering [1, 2, 18, 33], Image Captioning [29, 67], and Referring Expression [68].”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, neutral): “7 - - - AoANet [11] 37.”
XGPT: Cross-modal Generative Pre-Training for Image Captioning (2020, positive): “5 - - - AoANet [24] - - - - 119.”