Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

By Jiasen Lu, Caiming Xiong, Devi Parikh, Richard Socher. 2016

1,621 citations recorded by the source corpus.

Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as the and of. Other words that may seem visual can often be predicted reliably just from the language model e.g., sign after behind a red stop or phone following talking on a cell. In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, methodology): “For image captioning, Lu et al. [2016] train a visual sentinel with a pretrained CNN and ﬁne-tune the model with a smaller learning rate.”
Scaling Autoregressive Video Models (2019, background): “3, such that the attention matrix A is masked elementwise with (1 − I ) to allow for not attending to any element, similar to sentinel attention (Lu et al., 2017).”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “Architectures: We explore two auto-regressive architectures, namely a Long Short-Term Memory (LSTM) [15] with spatial attention-based model [31] and a transformerbased (TF) one [47].”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Image captioning models are increasingly being developed as a more complex way of labeling images [69, 50, 76, 48, 3, 33].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
nocaps: novel object captioning at scale (2019, background): “Recent progress in image captioning, the task of generating natural language descriptions of visual content [9,10,16, 17,41,44], can be largely attributed to the publicly available large-scale datasets of image-caption pairs [5,14,48] as well as steady modeling improvements [4, 24, 35, 46].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “anisms [21], the visual representation has turned into a time-varying vector extracted from a grid of CNN features using the hidden state of the language model as the query [22], [23], [6], [24], [8].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “in diﬀerent contexts. Attention networks also proved to be a useful tool in visual question answering applications, which require a joint image-text understanding to answer a question about the image [103; 64; 102; 65]. Yang et al. [105] propose a Stacked Attention Network (SAN) that uses two attent”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, background): “A visual sentinel is introduced in [23] to determine, at each decoding step, if it is important to attend the visual features.”
Neural Baby Talk (2018, methodology): “It is drawn from the language model , which is associated with a “default” sentinel “region” r̃ obtained from the language model [27] (discussed in Sec.”
Exploring Visual Relationship for Image Captioning (2018, methodology): “Moreover, SCST, ADP-ATT and Up-Down lead to a large performance boost over LSTM, which directly encodes image as one global representation.”
Auto-Encoding Scene Graphs for Image Captioning (2018, neutral): “plit, and a competitive 125:5 CIDEr-D (c40) on the ofﬁcial server even compared to other ensemble models. 1. Introduction Modern image captioning models employ an end-to-end encoder-decoder framework [27,28,2,26], i.e., the encoder encodes an image into vector representations and then the decoder de”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “On the image encoding side, instead, single-layer attention mechanisms have been adopted to incorporate spatial knowledge, initially from a grid of CNN features [45, 26, 50], and then using image regions extracted with an object detector [4, 29, 27].”
Boosting Image Captioning with Attributes (2016, neutral): “1 MetaMind/VT GT [15] 74.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Lu et al. [88] proposed another attention-based image captioning method.”
Guide Me: Interacting with Deep Networks (2018, background): “Most works focus on the combination of CNN and RNN models, often building attention mechanisms [3, 42, 59, 60].”