Convolutional Image Captioning
By Jyoti Aneja, Aditya Deshpande, Alexander Gerhard Schwing. 2017
401 citations recorded by the source corpus.
Image captioning is an important task, applicable to virtual assistants, editing tools, image indexing, and support of the disabled. In recent years significant progress has been made in image captioning, using Recurrent Neural Networks powered by long-short-term-memory (LSTM) units. Despite mitigating the vanishing gradient problem, and despite their compelling ability to memorize dependencies, LSTM units are complex and inherently sequential across time. To address this issue, recent work has shown benefits of convolutional networks for machine translation and conditional image generation [9, 34, 35]. Inspired by their success, in this paper, we develop a convolutional image captioning technique. We demonstrate its efficacy on the challenging MSCOCO dataset and demonstrate performance on par with the LSTM baseline [16], while having a faster training time per number of parameters. We also perform a detailed analysis, providing compelling reasons in favor of convolutional language generation approaches.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Yoshua Bengio
- Quoc V. Le
- Dacheng Tao
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Percy Liang
- Matteo Stefanini
- Giuseppe Fiameni
- Louis-philippe Morency
- Jiebo Luo
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Giuseppe Fiameni
- Idan Schwartz
- Mohammed Bany Muhammad
- Mohammed Yeasin
Citations in the corpus
- SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “[27] have defined a fully-convolutional captioning model using one-dimensional convolution and attention over image features.”
- Meshed-Memory Transformer for Image Captioning (2019, background): “Multi-level encodings of image regions are connected to a language decoder through a meshed and learnable connectivity. the last few years, with the exception of the investigation of Convolutional language models [5], which however did not become a leading choice.”
- A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Aneja et al. [5] proposed a convolutional architecture for the task of image captioning.”
- NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, background): “as et al., 2019; Razavi et al., 2019), music synthesis (Dhariwal et al., 2020), speech generation (Oord et al., 2016; Ping et al., 2020), image captioning (Aneja et al., 2019; Deshpande et al., 2019; Aneja et al., 2018), semi-supervised learning (Kingma et al., 2014; Izmailov et al., 2020), and repr”
- Eigen-CAM: Class Activation Map using Principal Components (2020, methodology): “The widely used convolutional neural network (CNN ) based architectures in DL are proven to be the best in learning representations and solving complex computer vision and general artificial intelligence problems such as image classification [4], [5], object localization [6]–[8], semantic segmentati”
- ContCap: A comprehensive framework for continual image captioning (2019, background): “From the introduction of Neural Image Caption Generator (NIC) [29], various advanced techniques have been proposed to ameliorate image captioning [19][4][1].”
- Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models (2021, background): “The topic of compositional learning has been extensively studied in V&L tasks including visual question answering (VQA) [3], image captioning [1, 2] and video retrieval [41].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, neutral): “[108], which uses convolutions as a language model.”
- Factor Graph Attention (2019, methodology): “Image captioning [36, 56, 24, 50, 7, 4, 13] was an instrumental language+vision task, followed by visual question answering [33, 42, 25, 34, 41, 5, 15, 59, 23, 3, 9, 14, 46, 55, 42, 54, 39, 40, 43] and visual question generation [41, 38, 22, 49, 28, 6].”
- From Show to Tell: A Survey on Image Captioning (2021, methodology): “A worth-to-mention approach is that proposed by Aneya et al. [88], which uses convolutions as a language model.”
- ContCap: A scalable framework for continual image captioning (2019, background): “Since the introduction of the Neural Image Caption Generator (NIC) [4], various powerful techniques have been proposed to enhance image captioning [5], [6], [7].”
- Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (2019, methodology): “creatively used by research communities centered around tasks such as multimodal language analysis [58], sentiment analysis [46, 31], emotion recognition [29], personality traits recognition [56], image captioning [5, 4, 34], multimedia description [50, 51, 62], and video comprehension [14, 32, 19, ”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, neutral): “, 2018) 3 7 Convolutional Attention (Aneja et al., 2018) 3 7”
- Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “• Baseline S2S uses the ConvCap [2], which has the same network architecture with the POS tag generator in our method except that the last 1D convolutional layer uses the attention mechanism to directly generate captions.”
- Big but Imperceptible Adversarial Perturbations via Semantic Manipulation (2019, methodology): “Recently, Janeja et al. proposed a convolutional based cap-tioning model for a fast and accurate caption generation [2].”
- RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, background): “[3] explored the convolutional language model in image captioning model.”
- Know More Say Less: Image Captioning Based on Scene Graphs (2019, background): “[43] propose a convolutional image captioning approach, which operates over all words in parallel.”
- Entangled Transformer for Image Captioning (2019, methodology): “Afterward, Aneja et al. [3] adapt this model to image captioning.”
- ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “On the text modeling side of the problem, language models (LMs) have also advanced with the development of LSTMs [16, 69], CNNs [4] and Transformers [27, 29, 47].”