Image Captioning with Semantic Attention

By Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, Jiebo Luo. 2016

1,793 citations recorded by the source corpus.

Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “[43] introduce image-level labels or attributes to improve image-level visual representations.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “As hardware acceleration grew more powerful and neural nets proved themselves in image understanding [41, 101, 78], interest intensified: autoregressive models were developed that could take samples of handwriting and complete them [28], and new convolutional network designs led to good results in s”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Image captioning models are increasingly being developed as a more complex way of labeling images [69, 50, 76, 48, 3, 33].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Joint visual-text models have become increasingly popular as they enable a wide suite of downstream tasks, including text-to-visual retrieval [36, 38, 43, 66], visual captioning [29, 65, 73], and visual question and answering [4, 33].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, background): “.0 LRCN [9] 71.8 89.5 54.8 80.4 40.9 69.5 30.6 58.5 24.7 33.5 52.8 67.8 92.1 93.4 17.7 59.9 Hard-Attention [52] 70.5 88.1 52.8 77.9 38.3 65.8 27.7 53.7 24.1 32.2 51.6 65.4 86.5 89.3 17.2 59.8 ATT-FCN [56] 73.1 90.0 56.5 81.5 42.4 70.9 31.6 59.9 25.0 33.5 53.5 68.2 94.3 95.8 18.2 63.1 Review Net [54]”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “olve video VQA tasks. There are very few approaches designed speciﬁcally to solve video VQA. Yu et al. [41] used LSTMs to represent both videos and QA pairs and adopted a semantic attention mechanism [39] on both input word representation and output word prediction. We also use LSTMs to represent bo”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Inspired by previous work [29, 27, 30, 15, 24], we add an attribute prediction branch in our subject module.”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “In contrast to earlier video captioning papers, which are based on models like hidden Markov models and ontologies [39, 6], recent work on captioning is dominated by deep neural network-based methods [32, 34, 37, 43, 36, 26].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “anisms [21], the visual representation has turned into a time-varying vector extracted from a grid of CNN features using the hidden state of the language model as the query [22], [23], [6], [24], [8].”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “In addition, th WN mechanism borrows ideas from attentive object understanding [31, 5, 95] in vision.”
Global-and-local attention networks for visual recognition (2018, methodology): “In this work, spatial and feature-based attention are combined in a single learned mask used to modulate feature representations, as is typically done in state-of-the-art systems [4–7].”
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives (2017, background): “Others suggest that rich visualizations or text supplied along with a decision, as is often done in deep learning for computer vision [16,5,6], oﬀer suﬃcient information to draw an explanation of why a particular decision was reached.”
Evolving Deep Neural Networks (2017, background): “Deep learning has recently provided state-of-the-art performance in image captioning, and several diverse architectures have been suggested [27, 46, 47, 49, 51].”
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning (2016, background): “For emb-gLSTM, it utilizes correlation between image and its description as gloabl semantic information, and for ATT it utilizes visual concepts corresponded words as semantic information.”
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning (2016, methodology): “We further compare our method with DeepVS [11], Hard-Attention [30] and recently proposed ATT [34], ERD [32] and best performed method (LSTM-A5) of MSM [33].”
Neural Baby Talk (2018, methodology): “Method BLEU1 BLEU4 METEOR CIDEr SPICE Hard-Attention [46] 66.9 19.9 18.5 - - ATT-FCN [50] 64.7 23.0 18.9 - - Adaptive [27] 67.7 25.1 20.4 53.1 14.5 NBT 69.0 27.1 21.7 57.5 15.6 NBToracle 72.0 28.5 23.1 64.8 19.6 Table 1.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “In vision, language is also used in captioning where the task is to generate a descriptive caption of an image or a video [10], [25], [26], [27].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “With the prevalence of deep learning [17] in computer vision, the dominant paradigm in modern image captioning is sequence learning methods [7,34,37,38,39,40] which utilize CNN plus RNN model to generate novel sentences with flexible syntactical structures.”
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (2016, methodology): “Second, the approach of [34] relies on the classifiers that are separately learned from external datasets, whereas our approach is learnable end-toend with only training data of captioning.”