Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
By Ke Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio. 2015
10,867 citations recorded by the source corpus.
Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO.
Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Kyunghyun Cho
- Luke S. Zettlemoyer
- Yejin Choi
- Richard Socher
- Percy Liang
In Multimodal Learning
Citations in the corpus
- Structured Attention Networks (2017, background): “…learning toolkit, contributing to impressive results in neural machine translation (Bahdanau et al., 2015; Luong et al., 2015), image captioning (Xu et al., 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al., 2015; Sukhbaatar et al., 2015),…”
- Can Active Memory Replace Attention? (2016, methodology): “In the extreme case, also known as hard attention [8], one of the memory elements is selected and the selection is trained using the REINFORCE algorithm (since this is not differentiable) [11].”
- Recurrent Neural Network Grammars (2016, methodology): “…baselines have been obtained using RNNs (Zaremba et al., 2015; Mikolov et al., 2010) as well as in various conditional language modeling tasks such as machine translation (Bahdanau et al., 2015), image caption generation (Xu et al., 2015), and dialogue generation (Wen et al., 2015).”
- Multi-task Sequence to Sequence Learning (2015, background): “Recent work such as (Xu et al., 2015; Jean et al., 2015a; Luong et al., 2015a; Vinyals et al., 2015a) has found that it is crucial to empower seq2seq models with the attention mechanism.”
- Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “The latter can be viewed as an interesting blend between the hard and soft attention models proposed in (Xu et al., 2015): it is computationally less expensive than the global model or the soft attention; at the same time, unlike the hard attention, the local attention is differentiable, making it…”
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “, 2014), HTML tables (Pasupat and Liang, 2015), and large (and noisy) corpora (Sawant and Chakrabarti, 2013; Joshi et al., 2014; Xu et al., 2015).”
- Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “Xu et al. (2015) used visual attention to produce higher-quality and more interpretable image captioning systems.”
- Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “In this respect our approach is similar to Xu et al. (2015), who apply a coverage-like method to image cap-tioning, and Chen et al. (2016), who also incorporate a coverage mechanism (which they call ‘dis-traction’) as described in equation (11) into neural summarization of longer text.”
- Quantifying Attention Flow in Transformers (2020, background): “…weights is the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark…”
- Exploring Self-Attention for Image Recognition (2020, neutral): “This has inspired applications of self-attention and related ideas to image recognition [5, 34, 15, 14, 45, 46, 13, 1, 27], image synthesis [43, 26, 2], image captioning [39, 41, 4], and video prediction [17, 35].”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Attention shows its ability of modeling long range interactions in language modeling [83,88], speech recognition [22,11], and neural captioning [92].”