Self-Critical N-Step Training for Image Captioning

By Junlong Gao, Shiqi Wang, Shanshe Wang, Siwei Ma, Wen Gao. 2019

63 citations recorded by the source corpus.

Existing methods for image captioning are usually trained by cross entropy loss, which leads to exposure bias and the inconsistency between the optimizing function and evaluation metrics. Recently it has been shown that these two issues can be addressed by incorporating techniques from reinforcement learning, where one of the popular techniques is the advantage actor-critic algorithm that calculates per-token advantage by estimating state value with a parametrized estimator at the cost of introducing estimation bias. In this paper, we estimate state value without using a parametrized value estimator. With the properties of image captioning, namely, the deterministic state transition function and the sparse reward, state value is equivalent to its preceding state-action value, and we reformulate advantage function by simply replacing the former with the latter. Moreover, the reformulated advantage is extended to n-step, which can generally increase the absolute value of the mean of reformulated advantage while lowering variance. Then two kinds of rollout are adopted to estimate state-action value, which we call self-critical n-step training. Empirically we find that our method can obtain better performance compared to the state-of-the-art methods that use the sequence level advantage and parametrized estimator respectively on the widely used MSCOCO benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Considering the training loss term, earlier works adopt the effective cross-entropy, while contemporary methods also apply self-critical sequence training [15, 32, 45].”
From Show to Tell: A Survey on Image Captioning (2021, methodology): “Finally, the most widely adopted reinforcement learning-based strategy [69], [105], [106], introduced by Rennie et al. [27], entails using the CIDEr score as reward, as it correlates well with human judgment [104].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Finally, the most widely adopted strategy [81], [123], [124], introduced by Rennie et al.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “9 in our case) is a very common practice during fine-tuning with RL, and is heavily adopted by the image-captioning literature [2], and very recently by on-the-fly FG-SBIR [1].”
Multimodal Transformer With Multi-View Visual Representation for Image Captioning (2019, background): “the subsequent image captioning researches [22], [28] and other related tasks that requires fine-grained image understanding [3], [5], [29].”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, methodology): “More recent methods consider the use of various deep learning apparatus such as GANs [17,31], deep reinforcement learning [10,38], and transformers [8,12].”
A Better Variant of Self-Critical Sequence Training (2020, methodology): “It is still a “Self-Critical” baseline because the critic is still from itself: its other sampling results, instead of the greedy decoding result. Our variant can also been seen as a special case of (Gao et al., 2019). While SCST is T-step-maxpro, ours is Tstep-sample, with the naming in (Gao et al.”