Improved Image Captioning via Policy Gradient optimization of SPIDEr

By Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, Kevin P. Murphy. 2016

497 citations recorded by the source corpus.

Current image captioning methods are usually trained via maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “A large number of DNN models for image caption generation have been trained and evaluated using COCO captions (Vinyals et al., 2015a; Fang et al., 2015; Xu et al., 2015; Ranzato et al., 2015; Yang et al., 2016; Liu et al., 2017; Ding and Soricut, 2017).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Additionally, we also train with task-specific rewards such as CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al., 2017) which is a just a linear combination of both with equal weights.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, neutral): “Reinforcement learning enables model optimization with non-differentiable evaluation metrics [14, 45, 61, 64].”
nocaps: novel object captioning at scale (2019, methodology): “We expect this subset to be the most challenging and models trained only on COCO data are likely to make ‘embarrassing errors’ [23] on this subset, reflecting the current performance of COCO trained models in the wild.”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “[16], [17], [7] with a visual feature extractor for conditioning”
Adversarial Ranking for Language Generation (2017, methodology): “Importantly, different from the policy gradients methods in other works [4, 20, 35], our method replaces the simple binary outputs with a ranking system based on multiple sentences, which can better reflect the quality of the imitate sentences and facilitate effective training of the generator G.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “On the training side, while initial methods were based on a time-wise cross-entropy training, a notable achievement has been made with the introduction of Reinforcement Learning, which enabled the use of non-differentiable caption metrics as optimization objectives [33, 31, 25].”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “(10) with the intermediate reward to accelerate the training speed and effectiveness (Liu et al., 2017).”
Reinforced Video Captioning with Entailment Rewards (2017, methodology): “1 Liu et al. (2016b) and Rennie et al. (2016) improve upon this using Monte Carlo roll-outs and a test inference baseline, respectively.”
Evaluation of Text Generation: A Survey (2020, methodology): “, 2014), recent studies have shown that metrics that include semantic content matching such as spice can correlate better with human judgments (Anderson et al., 2016; Liu et al., 2017).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Liu et al. [85] introduced a new captions evaluation metric that is a good choice by human raters.”
Robust Text-to-SQL Generation with Execution-Guided Decoding (2018, methodology): “…Several recently introduced techniques use reinforcement learning to incorporate sequence-level objectives into training, such as BLEU in coherent text generation (Bosselut et al. 2018), semantic coherence in dialogue generation (Li et al. 2016), and SPIDEr in image captioning (Liu et al. 2017).”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Recently, deep reinforcement learning has attracted growing attention in the ﬁeld of visual captioning (Ren et al., 2017; Rennie et al., 2017; Liu et al., 2017; Wang et al., 2018b).”
Interactive Image Generation Using Scene Graphs (2019, unknown): “Scene graphs are a powerful structured representation for both images and language; they have been used for semantic image retrieval (Johnson et al., 2015) and for evaluating (Anderson et al., 2016) and improving (Liu et al., 2017) image captioning.”