Stack-Captioning: Coarse-to-Fine Learning for Image Captioning

By Jiuxiang Gu, Jianfei Cai, G. Wang, Tsuhan Chen. 2017

194 citations recorded by the source corpus.

The existing image captioning approaches typically train a one-stage sentence decoder, which is difficult to generate rich fine-grained descriptions. On the other hand, multi-stage image caption model is hard to train due to the vanishing gradient problem. In this paper, we propose a coarse-to-fine multi-stage prediction framework for image captioning, composed of multiple decoders each of which operates on the output of the previous stage, producing increasingly refined image descriptions. Our proposed learning approach addresses the difficulty of vanishing gradients during training by providing a learning objective function that enforces intermediate supervisions. Particularly, we optimize our model with a reinforcement learning approach which utilizes the output of each intermediate decoder's test-time inference algorithm as well as the output of its preceding decoder to normalize the rewards, which simultaneously solves the well-known exposure bias problem and the loss-evaluation mismatch problem. We extensively evaluate the proposed approach on MSCOCO and show that our approach can achieve the state-of-the-art performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, background): “For statistical language modeling, the input typically consists of incomplete sequences of words rather than complete sentences [304, 305].”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Specifically, we compared our methods with SCST [38], StackCap [9], Up-Down [2], LSTMA [54], GCN-LSTM [53], and CAVP [27].”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “Scene graph provides a structured representation of an image that can support a wide range of high-level visual tasks, including image captioning [12, 14, 13, 43], visual question answering [36, 38, 47], image retrieval [11, 21], and image generation [20].”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “[37] devised a coarse-to-fine multistage framework using a sequence of LSTM decoders, each operating on the output of the previous one to produce increasingly refined captions.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “On the same line, Gu et al. [49] devised a coarse-to-fine multi-stage framework using a sequence of LSTM decoders, each operating on the output of the previous one to produce increasingly refined captions.”
Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing (2020, background): “Also, the idea that ‘training a stronger model at the second time’ is related to the coarse-to-fine learning approaches [14, 49] which often adopted iteration for larger improvements.”
Unpaired Image Captioning by Language Pivoting (2018, methodology): “Recently, the loss-evaluation mismatch problem has been well-addressed in sequence prediction tasks [29, 7, 31, 32, 8].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “, 2018) 3 3 Stack Captioning (Gu et al., 2018) 3 3”
Learning to Collocate Neural Modules for Image Captioning (2019, result): “All of StackCap, CAVP, and RFNet use wider encoders or deeper decoders, while they do not design different modules.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, background): “, ‘the,’ ‘and’) [45], generating fine, novel and diverse sentences [7,24,70], and incorporating information from different semantic taggers [12, 33].”
VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions (2018, methodology): “Attention mechanism is firstly used in machine translation [3] and then is brought into the vision-to-language tasks [29,32,28,31,18,15,19,33,10,9,30].”
Discriminability Objective for Training Descriptive Captions (2018, background): “Within this generic framework, many efforts [7, 10, 11, 8, 12, 13, 14, 15, 16] explored different encoder-decoder structures, including attention-based models.”
A Survey of Deep Learning-Based Object Detection (2019, background): “issue, multimodal embedding, encoderdecoder frameworks, attention mechanism [75] [213], and reinforcement learning [214] [215] are widely adopted in this field.”