Deep Reinforcement Learning-Based Image Captioning with Embedding Reward

By Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, Li-Jia Li. 2017

339 citations recorded by the source corpus.

Image captioning is a challenging problem owing to the complexity in understanding the image content and diverse ways of describing it in natural language. Recent advances in deep neural networks have substantially improved the performance of this task. Most state-of-the-art approaches follow an encoder-decoder framework, which generates captions using a sequential recurrent prediction model. However, in this paper, we introduce a novel decision-making framework for image captioning. We utilize a policy network and a value network to collaboratively generate captions. The policy network serves as a local guidance by providing the confidence of predicting the next word according to the current state. Additionally, the value network serves as a global and lookahead guidance by evaluating all possible extensions of the current state. In essence, it adjusts the goal of predicting the correct words towards the goal of generating captions similar to the ground truth captions. We train both networks using an actor-critic reinforcement learning model, with a novel reward defined by visual-semantic embedding. Extensive experiments and analyses on the Microsoft COCO dataset show that the proposed framework outperforms state-of-the-art approaches across different evaluation metrics.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Note that more sophisticated baselines based on neural adaptation [22] or self-critic [23] can be also employed.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Many image captioning methods [26, 39, 61, 112, 119, 126, 135, 144, 149, 151, 156] use the dataset in their experiments.”
Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks (2018, methodology): “Following the existing visual-semantic embedding works [Ren et al., 2017], we choose the reward function based on the embedding similarity between the ground-truth answer a and the generated answer â, given by Ra(â) = ‖h − h‖(2).”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “…word appearance prediction; [19] uses Conditional Variational Autoencoder(CVAE) and optimize over evidence lower bound(ELBO); [20, 6, 21, 22, 23, 24] applied Reinforcement Learning algorithms on captioning, so that the models can be optimized directly on the non-differentiable rewards like…”
Resources and End-to-End Neural Network Models for Arabic Image Captioning (2020, background): “Ren et al. (Ren et al., 2017) proposed a novel RL based image cap-tioning that consists of two networks: the policy network predicts the next word based on the current state, and the value network guides the policy network by evaluating its reward.”
Discourse-Aware Neural Rewards for Coherent Text Generation (2018, background): “Most similar to our work is work on using neural and embedding rewards to improve dialogue (Li et al., 2016), image captioning (Ren et al., 2017), simpliﬁcation (Zhang and Lapata, 2017), and paraphrase generation (Li et al., 2017).”
Deep Multimodal Representation Learning: A Survey (2019, background): “Conversely, an exemplary application of deep reinforcement learning during decoding is image captioning [94], [95].”
Part-Activated Deep Reinforcement Learning for Action Prediction (2018, background): “[30] presented a novel decision-making framework for image captioning utilizing a policy network and a value network.”
Visual to Text: Survey of Image and Video Captioning (2019, background): “and a value network [139], the self-critical sequence training model [140], reinforced video using captioning entailmentenhanced reward [141], etc.”
Video Captioning via Hierarchical Reinforcement Learning (2017, positive): “Naturally, the variance could be reduced by adding a baseline [13, 23] or even an actor-critic method that trained an additional critic to estimate the value of each generated word [1, 22, 43].”
Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards (2017, methodology): “In [23], Ren et al . use a policy network and a value network to collaboratively generate image cap-tions, while different optimization methods for RL in image captioning are explored in [17] and [24], called SPIDEr and self-critical sequence training.”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, methodology): “Motivated by the success of neural machine translation (NMT) [1] and neural image captioning [34, 54], deep neural network models with encoder-decoder pipeline have been applied to video captioning recently and achieved excellent performance [9, 29, 45, 51].”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “Examples include applying expert demonstration to build RL solutions to Spoken Dialogue Systems [151], VQA [148]; or building shaped rewards for Sequence Generation [152], Spoken Dialogue Systems [64],QA [65], [153], and Image Caption [149], or transferring policies for Structured Prediction [154] a”
Reconstruction Network for Video Captioning (2018, background): “Previous work predominantly focused on describing still images with natural language [15, 41, 42, 28, 13, 5].”
Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, background): “There have been very few works which have used unsupervised (Feng et al., 2018) or reinforcement learning (Ren et al., 2017) for captioning images.”
Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, background): “[38] propose an actor-critic deep reinforcement learning model with an embedding reward for image captioning.”