Image Captioning with Deep Bidirectional LSTMs

By Cheng Wang, Haojin Yang, Christian Bartz, Christoph Meinel. ACM Multimedia, 2016

299 citations recorded by the source corpus.

This work presents an end-to-end trainable deep bidirectional LSTM (Long-Short Term Memory) model for image captioning. Our model builds on a deep convolutional neural network (CNN) and two separate LSTM networks. It is capable of learning long term visual-language interactions by making use of history and future context information at high level semantic space. Two novel deep bidirectional variant models, in which we increase the depth of nonlinearity transition in different way, are proposed to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale and vertical mirror are proposed to prevent overfitting in training deep models. We visualize the evolution of bidirectional LSTM internal states over time and qualitatively analyze how our models "translate" image to sentence. Our proposed models are evaluated on caption generation and image-sentence retrieval tasks with three benchmark datasets: Flickr8K, Flickr30K and MSCOCO datasets. We demonstrate that bidirectional LSTM models achieve highly competitive performance to the state-of-the-art results on caption generation even without integrating additional mechanism (e.g. object detection, attention model etc.) and significantly outperform recent methods on retrieval task

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/26e425781e4090abfae65b5d68eac72282dd2e31

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FOIL it! Find One mismatch between Image and Language caption (2017, methodology): “For the latter, we use the multimodal bi-directional LSTM, proposed in (Wang et al., 2016), and adapted for our tasks.”
Twin Networks: Matching the Future for Sequence Generation (2017, methodology): “Due to the lack of standardized split of training, validation and test data, we follow Karpathy’s split (Karpathy & Fei-Fei, 2015; Xu et al., 2015; Wang et al., 2016).”
Predicting Visual Features From Text for Image and Video Caption Retrieval (2017, background): “While RNN and its LSTM or GRU variants have demonstrated promising results for generating visual descriptions [49]–[52], they tend to be over-sensitive to word orders by design.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Recently, pointing LSTM is designed to point the novel objects (Li et al., 2019) by balancing generation and copying words.”
Deep learning for visual understanding (2017, unknown): “Following most recent works [266, 302, 306, 312] , we use 5000 images as validation set to tune hyper-parameters, and another 5000 images as test set to report results.”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, unknown): “Deep learning-based encoder-decoder architecture [4, 9, 17, 18, 21, 25–27, 31, 34, 41, 48, 52–54, 56] has shown its effectiveness in video captioning.”
Text Augmentation Using BERT for Image Captioning (2020, unknown): “The most of them, such as [33,34], use standard images augmentation methods described above.”
Fluency-Guided Cross-Lingual Image Captioning (2017, background): “[37] propose a deeper bidirectional variant of Long Short Term Memory (LSTM) to take both history and future context into account in image captioning.”
Know More Say Less: Image Captioning Based on Scene Graphs (2019, unknown): “Recently, prevailing methods based on neural networks have adopted convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to address this task [3]–[6], [20], [22], [24], [28], [37]– [39].”
Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction (2016, methodology): “[37], linear CCA with fisher vector pooling over word vectors by Klein et al.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
Watch What You Just Said: Image Captioning with Text-Conditional Attention (2016, background): “Recent literature in image captioning is dominated by neural network-based methods [6, 31, 33, 38].”