Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition

By Yufei Wang, Zhe L. Lin, Xiaohui Shen, Scott D. Cohen, Garrison W. Cottrell. 2017

107 citations recorded by the source corpus.

Recently, there has been a lot of interest in automatically generating descriptions for an image. Most existing language-model based approaches for this task learn to generate an image description word by word in its original word order. However, for humans, it is more natural to locate the objects and their relationships first, and then elaborate on each object, describing notable attributes. We present a coarse-to-fine method that decomposes the original image description into a skeleton sentence and its attributes, and generates the skeleton sentence and attribute phrases separately. By this decomposition, our method can generate more accurate and novel descriptions than the previous state-of-the-art. Experimental results on the MS-COCO and a larger scale Stock3M datasets show that our algorithm yields consistent improvements across different evaluation metrics, especially on the SPICE metric, which has much higher correlation with human ratings than the conventional metrics. Furthermore, our algorithm can generate descriptions with varied length, benefiting from the separate control of the skeleton and attributes. This enables image description generation that better accommodates user preferences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Generation

Citations in the corpus

ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works have used LSTM variants [8, 38, 39], while recent works [16, 26] adopted the improved transformer architecture [36].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Wang et al. [148] proposed a method that locates the objects and their interactions first and then identifies and extracts the relevant attributes to generate image captions.”
Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis (2019, background): “Furthermore, following the evaluations in [31], the ratio of the unique sentences and the ratio of the novel sentences were calculated”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “[47] proposed to generate a caption from coarse central aspects to finer attributes”
Discriminability Objective for Training Descriptive Captions (2018, background): “Within this generic framework, many efforts [7, 10, 11, 8, 12, 13, 14, 15, 16] explored different encoder-decoder structures, including attention-based models.”
CNN+CNN: Convolutional Decoders for Image Captioning (2018, background): “Similarly, [32] applies a skeleton LSTM to generate the skeleton words of the sentence, and an attribute LSTM to generate adjective words that modify the skeleton words, which is a coarse-to-fine model.”