Reinforced Video Captioning with Entailment Rewards

By Ramakanth Pasunuru, Mohit Bansal. 2017

118 citations recorded by the source corpus.

Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “This includes REINFORCE (Williams, 1992) for Machine Translation (Ranzato et al., 2016), actor critic for Abstractive Summarization (Paulus et al., 2018), Image-to-Text (Liu et al., 2016), Dialogue Generation (Li et al., 2016b), and Video Captioning (Pasunuru & Bansal, 2017).”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “We instantiate the captioning model by adapting the model architectures from the state-of-the-art video captioning methods [43, 61].”
Multi-Reward Reinforced Summarization with Saliency and Entailment (2018, methodology): “Pasunuru and Bansal (2017) used entailment-corrected phrase-matching metrics (CIDEnt) to improve the task of video captioning; we instead directly use the entailment knowledge from an entailment scorer and its multi-sentence, length-normalized extension as our ‘En-tail’ reward, to improve the task…”
Deep Communicating Agents for Abstractive Summarization (2018, neutral): “Reinforcement Learning (RL) Loss Policy gradient methods can directly optimize discrete target evaluation metrics such as ROUGE that are non-differentiable (Paulus et al., 2018; Jaques et al., 2017; Pasunuru and Bansal, 2017; Wu et al., 2016).”
A Distributional Approach to Controlled Text Generation (2020, methodology): “This includes REINFORCE (Williams, 1992a) for Machine translation (MT) Ranzato et al. (2016), actor critic for Abstractive Summarization (Paulus et al., 2018), Imageto-Text Liu et al. (2016b), Dialogue Generation Li et al. (2016b), and Video Captioning (Pasunuru & Bansal, 2017).”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “…tasks ranging from translation (Choshen et al., 2019), summarisation (Paulus et al., 2018), story generation (Tambwekar et al., 2019), video captioning (Pasunuru and Bansal, 2017), dialogue (Jaques et al., 2019), to code generation (Korbak et al., 2021) and LM debiasing (Khalifa et al., 2021).”
Generating Diverse and Natural 3D Human Motions from Text (2022, background): “…has been fundamentally changed in the deep learning era, where we witness significant performance boosts by adopting a variety of powerful techniques including RNNs [30,40], transformer [18], attentive context modeling [41], memory network [28, 46], GANs [26], and reinforcement learning [19,27].”
Automatic Document Sketching: Generating Drafts from Analogous Texts (2021, methodology): “runs the risk of compromising the language model (Paulus et al., 2018; Pasunuru and Bansal, 2017), we use a mixed loss as follows:”
Visual to Text: Survey of Image and Video Captioning (2019, background): “and a value network [139], the self-critical sequence training model [140], reinforced video using captioning entailmentenhanced reward [141], etc.”
Knowledge-Guided Agent-Tactic-Aware Learning for StarCraft Micromanagement (2018, background): “…learning (DRL) [Mnih et al. , 2015; Li, 2017; Silver et al. , 2016; Lillicrap et al. , 2015; Levine et al. , 2016; Mirowski et al. , 2017; Pasunuru and Bansal, 2017; Li et al. , 2017; Silver et al. , 2017] has at-tracted considerable attention in the ﬁelds of artiﬁcial intelligence and…”
Discourse-Aware Neural Rewards for Coherent Text Generation (2018, methodology): “Consequently, it is possible that generated sequences are no longer readable (Pasunuru and Bansal, 2017; Paulus et al., 2018).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “It is further approached from the reinforcement learning by providing entailment rewards (Pasunuru & Bansal, 2017b) and to address the description generation for multiple fine-grained actions (Wang et al., 2018b).”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “The same observations can be observed if using (I3D, M) pair as the feature representations in Table 3, which proves that the POS information provides a global view on the potential syntactic structure of its language descriptions and thereby further improves the performance of video captioning.”