Boosting Image Captioning with Attributes

By Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei. 2016

656 citations recorded by the source corpus.

Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. Particularly, the learning of attributes is strengthened by integrating inter-attribute correlations into Multiple Instance Learning (MIL). To incorporate attributes into captioning, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework shows clear improvements when compared to state-of-the-art deep models. More remarkably, we obtain METEOR/CIDEr-D of 25.5%/100.2% on testing data of widely used and publicly available splits in [10] when extracting image representations by GoogleNet and achieve superior performance on COCO captioning Leaderboard.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Besides, researchers have explored to leverage semantic attributes [26,53], scene graphs [50], and graph convolutional networks [52] for captioning.”
Attention on Attention for Image Captioning (2019, background): “More recently, more complex information such as objects, attributes and relationships are integrated to generate better descriptions [50, 2, 49, 44].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, background): “However, these and other non-attention attribute detection approaches [55, 50] do not retain spatial information, treating the detected attributes as a bag of words.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Inspired by previous work [29, 27, 30, 15, 24], we add an attribute prediction branch in our subject module.”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “In contrast to earlier video captioning papers, which are based on models like hidden Markov models and ontologies [39, 6], recent work on captioning is dominated by deep neural network-based methods [32, 34, 37, 43, 36, 26].”
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning (2016, methodology): “We further compare our method with DeepVS [11], Hard-Attention [30] and recently proposed ATT [34], ERD [32] and best performed method (LSTM-A5) of MSM [33].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “(4) LSTMA [39] integrates semantic attributes into CNN plus RNN captioning model for boosting image captioning.”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Specifically, we compared our methods with SCST [38], StackCap [9], Up-Down [2], LSTMA [54], GCN-LSTM [53], and CAVP [27].”
Improved Image Captioning via Policy Gradient optimization of SPIDEr (2016, methodology): “For example, [24, 28] use image taggers, [21] use object detectors, and [19] use face detection and landmark recognition.”
Grounded Video Description (2018, background): “Visual attention usually comes in the form of temporal attention [34] (or spatial-attention [32] in the image domain), semantic attention [13, 35, 36, 41] or both [19].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “A general approach of this category is to analyze the visual content of the image first and then generate image captions from the visual content using a language model [70, 152, 155, 156].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, neutral): “visual encoding [40], [43], [46], [47], [48], [49].”
Recurrent Fusion Network for Image Captioning (2018, methodology): “We compare our proposed RFNet with the state-of-the-art approaches on image captioning, including Neural Image Caption (NIC) [18], Attribute LSTM [23], LSTM-A3 [24], Recurrent Image Captioner (RIC) [46], Recurrent Highway Network (RHN) [47], Soft Attention model [12], Attribute Attention model [48],”
Visual to Text: Survey of Image and Video Captioning (2019, background): “have shown to be useful for assisting the architecture of visual encoder to text decoder [100], where the semantics are usually formulated as visual attributes [95], [98], [100].”
Entity-Enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding (2022, neutral): “As mentioned in previous methods [7], [51], [52], attribute information is essential to distinguish the object of the same category.”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “For example, [17, 18] add some auxiliary tasks like word appearance prediction; [19] uses Conditional Variational Autoencoder(CVAE) and optimize over evidence lower bound(ELBO); [20, 6, 21, 22, 23, 24] applied Reinforcement Learning algorithms on captioning, so that the models can be optimized…”
CNN+CNN: Convolutional Decoders for Image Captioning (2018, background): “Although this semantic attention significantly improves the performance, to some extent, it has a problem of prediction error accumulation along generated sequences [35].”
Know More Say Less: Image Captioning Based on Scene Graphs (2019, neutral): “Recently, prevailing methods based on neural networks have adopted convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to address this task [3]–[6], [20], [22], [24], [28], [37]– [39].”