Actor-Critic Sequence Training for Image Captioning

By Li Zhang, Flood Sung, Feng Liu, Tao A. Xiang, Shaogang Gong, Yongxin Yang, Timothy M. Hospedales. 2017

116 citations recorded by the source corpus.

Generating natural language descriptions of images is an important capability for a robot or other visual-intelligence driven AI agent that may need to communicate with human users about what it is seeing. Such image captioning methods are typically trained by maximising the likelihood of ground-truth annotated caption given the image. While simple and easy to implement, this approach does not directly maximise the language quality metrics we care about such as CIDEr. In this paper we investigate training image captioning methods based on actor-critic reinforcement learning in order to directly optimise non-differentiable quality metrics of interest. By formulating a per-token advantage and value computation strategy in this novel reinforcement learning based captioning model, we show that it is possible to achieve the state of the art performance on the widely used MSCOCO benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Considering the training loss term, earlier works adopt the effective cross-entropy, while contemporary methods also apply self-critical sequence training [15, 32, 45].”
Improving Abstraction in Text Summarization (2018, methodology): “mismatch between the optimization objective and the evaluation metrics by directly optimizing evaluation metrics. This approach has led to consistent improvements in domains such as image captioning (Zhang et al., 2017) and abstractive text summarization (Paulus et al.,2017). A recent approach to tr”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Zhang et al. [161] proposed an actor-critic reinforcement learning-based image captioning method.”
Video Captioning via Hierarchical Reinforcement Learning (2017, unknown): “Naturally, the variance could be reduced by adding a baseline [13, 23] or even an actor-critic method that trained an additional critic to estimate the value of each generated word [1, 22, 43].”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “For example, [17, 18] add some auxiliary tasks like word appearance prediction; [19] uses Conditional Variational Autoencoder(CVAE) and optimize over evidence lower bound(ELBO); [20, 6, 21, 22] applied Reinforcement Learning algorithms on image captioning, so that the models can be optimized directl”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “Finally, the most widely adopted reinforcement learning-based strategy [69], [105], [106], introduced by Rennie et al. [27], entails using the CIDEr score as reward, as it correlates well with human judgment [104].”
Self-Critical N-Step Training for Image Captioning (2019, methodology): “However, the value/baseline estimator was used in [17,21,29], which introduces estimation bias.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Finally, the most widely adopted strategy [81], [123], [124], introduced by Rennie et al.”
Learning a Deep Embedding Model for Zero-Shot Learning (2016, background): “Semantic space Existing ZSL methods differ in what semantic spaces are used: typically either attribute [9, 10, 35, 47], word vector [46, 11], or text description [37, 53].”
Deep Reinforcement Learning (2023, unknown): “To address these challenges, researchers have developed techniques s.a. reward shaping, where the reward function is designed to encourage desirable behavior, and curriculum learning, where the agent is trained on progressively more difficult tasks [25], [26].”