Image Caption Generation with Text-Conditional Semantic Attention

By Luowei Zhou, Chenliang Xu, Parker A. Koch, Jason J. Corso. 2016

44 citations recorded by the source corpus.

We propose a semantic attention mechanism for image caption generation, called text-conditional semantic attention, which allows the caption generator to automatically learn which parts of the image feature to focus on given previously generated text. To acquire text-related image features for our attention model, we also improve the guiding Long Short-Term Memory (gLSTM) structure by back-propagating the training loss though semantic guidance to fine-tune the CNN weights. In contrast to existing gLSTM methods, such as emb-gLSTM, our fine-tuned model enables guidance information to be more text-related. This also allows jointly learning of the image embedding, text embedding, semantic attention and language model with one simple network architecture in an end-to-end manner. We implement our model based on NeuralTalk2, an open-source image caption generator, and test it on MSCOCO dataset. We evaluate the proposed method with three metrics: BLEU, METEOR and CIDEr. The proposed methods outperform state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Tao Mei
3. Dacheng Tao
4. Yi Yang
5. Cordelia Schmid
6. Trevor Darrell
7. Xiaogang Wang
8. Andrew Zisserman

In Multimodal Learning

1. Tao Mei
2. Ting Yao
3. Yingwei Pan
4. Yehao Li
5. Luowei Zhou
6. Sanja Fidler
7. Jason J. Corso
8. Raquel Urtasun

In Natural Language Generation

1. Tao Mei
2. Jason J. Corso

Citations in the corpus

• Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “Then, we extract the frame-wise ResNet-34 feature (He et al. 2016), 3 pretrained on both ImageNet (Deng et al. 2009) and MSCOCO caption (Lin et al. 2014; Zhou et al. 2016).”
• Boosting Image Captioning with Attributes (2016, methodology): “(5) Sentence-Condition (SC) [39]: Sentence-condition exploits text-conditional semantic attention to generate semantic guidance for sentence generation by conditioning image features on current text content.”
• Towards Diverse and Natural Image Descriptions via a Conditional GAN (2017, background): “Many state-of-the-art frameworks [21, 29–32 , 35 ] for Here, I i and S i = ( w ) are the image and the corresponding descriptive sentence of the i -th sample, and n is the order of the Markov chain – the distribution of the current word depends on n preceding words.”