Self-Critical Sequence Training for Image Captioning

By Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, Vaibhava Goel. 2016

2,119 citations recorded by the source corpus.

Recently it has been shown that policy-gradient methods for reinforcement learning can be utilized to train deep end-to-end systems directly on non-differentiable metrics for the task at hand. In this paper we consider the problem of optimizing image captioning systems using reinforcement learning, and show that by carefully optimizing our systems using the test metrics of the MSCOCO task, significant gains in performance can be realized. Our systems are built using a new optimization approach that we call self-critical sequence training (SCST). SCST is a form of the popular REINFORCE algorithm that, rather than estimating a baseline to normalize the rewards and reduce variance, utilizes the output of its own test-time inference algorithm to normalize the rewards it experiences. Using this approach, estimating the reward signal (as actor-critic methods must do) and estimating normalization (as REINFORCE algorithms typically do) is avoided, while at the same time harmonizing the model with respect to its test-time inference procedure. Empirically we find that directly optimizing the CIDEr metric with SCST and greedy decoding at test-time is highly effective. Our results on the MSCOCO evaluation sever establish a new state-of-the-art on the task, improving the best result in terms of CIDEr from 104.9 to 114.7.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “In our model, we use the self-critical policy gradient training algorithm (Rennie et al., 2016).”
Persistent Anti-Muslim Bias in Large Language Models (2021, methodology): “These descriptions are generated by passing images through a pretrained image captioning model [11], and fine-tuning GPT-3 to transform the short captions into long descriptions, as shown in Fig.”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “Due to the non-differentiable sampling procedure, the gradient is approximated using a single sample with some variance reduction baseline b : We follow the effective SCST strategy (Rennie et al., 2017) to take the reward of greedy search result q g as the baseline, i.e., b = r ( q g ) .”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, positive): “re-training scheme. (iii) overall, Oscar is the best performer on both understanding and generation tasks. On the captioning task, we further ne-tune Oscar with self-critical sequence training (SCST) [31] to improve sequence-level learning. The only comparable VLP method for captioning is [47]. The ”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Various methods have been delivered for image captioning (Karpa-thy and Fei-Fei, 2015; Rennie et al., 2017; Anderson et al., 2018a; Luo et al., 2018; Luo, 2020).”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Rennie et al. (2017) proposed a method that avoids learning a separate critic.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “, 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et al.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In some cases our results likely trail the state of the art due in part to the fact that we simply optimise log-likelihood and do not make use of common task-specific metric optimisation tricks, such as CIDEr optimisation [64, 90] for COCO captioning, and fine-tuning on dense annotations for VisDial”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Considering the training loss term, earlier works adopt the effective cross-entropy, while contemporary methods also apply self-critical sequence training [15, 32, 45].”
Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “Particularly, in Tbl. 2 we see from Att2in to DiscCap and FC to Transformer, the AUCs slightly increases from 62.8±1.5 to 63.0±3.2 and 65.9±4.2 to 66.0±1.4, respectively.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Reinforcement learning enables model optimization with non-differentiable evaluation metrics [14, 45, 61, 64].”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “We further perform CIDEr optimization on COCO Cap-tions through Self-Critical Sequence Training (SCST) (Ren-nie et al. 2017), as in most of the recent image captioning literatures.”
Attention on Attention for Image Captioning (2019, methodology): “Then we directly optimize the non-differentiable metrics with Self-Critical Sequence Training [31] (SCST): LRL(θ) = −Ey 1:T∼pθ [r(y1:T )] (16) where the reward r(·) uses the score of some metric (e.g. CIDEr-D [36]).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “We comprehensively evaluate the captioning performance on the widely-used Karpathy split (Karpathy & Li, 2015) of COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014), the COCO test set, nocaps (Agrawal et al., 2019)2 which focuses on novel objects, TextCaps (Sidorov et al., 2020) which focuse”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “As the evaluation is single-shot, at test time we use greedy decoding [38].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “For fair comparison with recent work [33] we also report results optimized for CIDEr [42].”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “In contrast to earlier video captioning papers, which are based on models like hidden Markov models and ontologies [39, 6], recent work on captioning is dominated by deep neural network-based methods [32, 34, 37, 43, 36, 26].”