An Empirical Study of Language CNN for Image Captioning

By Jiuxiang Gu, G. Wang, Jianfei Cai, Tsuhan Chen. 2016

150 citations recorded by the source corpus.

Language models based on recurrent neural networks have dominated recent image caption generation tasks. In this paper, we introduce a language CNN model which is suitable for statistical language modeling tasks and shows competitive performance in image captioning. In contrast to previous models which predict next word based on one previous word and hidden state, our language CNN is fed with all the previous words and can model the long-range dependencies in history words, which are critical for image captioning. The effectiveness of our approach is validated on two datasets: Flickr30K and MS COCO. Our extensive experimental results show that our method outperforms the vanilla recurrent neural network based language models and is competitive with the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, background): “However, the frameworks in [308] and [307] are still under the recurrent framework, and the input window size of their network are of limited size.”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Modern image captioning models employ an end-to-end encoder-decoder framework [10, 30, 29, 2, 27], i.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Inspired by the above success of CNNs in sequence learning tasks, Gu et al. [51] proposed a CNN language model-based image captioning method.”
VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions (2018, background): “Attention mechanism is ﬁrstly used in machine translation [3] and then is brought into the vision-to-language tasks [29,32,28,31,18,15,19,33,10,9,30].”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Recently, convolutional networks [34, 9, 18] have proven their superior strength in modeling the time series data [60, 8, 61].”
Unpaired Image Captioning by Language Pivoting (2018, background): “Recently, Gu et al [9] propose a CNN-based image captioning model, which can explore both long-term and temporal information in word sequences for caption generation.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, unknown): “[11] 73.”
Entangled Transformer for Image Captioning (2019, background): “To get rid of the catastrophic forgetting in long-term memory, Gu et al. [15] introduce temporal CNN to impose the experienced semantic information at every step of the generation procedure.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “, 2017a) 7 7 Language CNN (Gu et al., 2017) 7 7 ConceptNet-NIC (Zhou et al.”
Multimodal Contrastive Training for Visual Representation Learning (2021, background): “More importantly, such multimodal information comes with higher-level abstract concepts, which offer the potential for drawing useful connections across different modalities [22, 31, 34, 20, 15].”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “Scene graph provides a structured representation of an image that can support a wide range of high-level visual tasks, including image captioning [12, 14, 13, 43], visual question answering [36, 38, 47], image retrieval [11, 21], and image generation [20].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Global CNN features were then employed in a large variety of image captioning models [30], [31], [32], [33], [34], [35], [36], [37].”
Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation (2018, background): “To ensure every information map be aware of global information, the outputs of RNN are concatnated , Hp = (h1p...h N p ) T , and refined with global information: Hp = Fg (Hp,Θg) + Hp (4) where Fg is a 1×1×N×N Conv and Θg is its parameters.”
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models (2017, background): “In recent years, a great deal of research has been devoted to bridge the heterogeneity gap between different modalities [15, 12, 16, 8, 38, 7, 6, 3, 34].”