Video Captioning via Hierarchical Reinforcement Learning

By Xin Eric Wang, Wenhu Chen, Jiawei Wu, Yuan-fang Wang, William Yang Wang. 2017

254 citations recorded by the source corpus.

Video captioning is the task of automatically generating a textual description of the actions in a video. Although previous work (e.g. sequence-to-sequence model) has shown promising results in abstracting a coarse description of a short video, it is still very challenging to caption a video containing multiple fine-grained actions with a detailed description. This paper aims to address the challenge by proposing a novel hierarchical reinforcement learning framework for video captioning, where a high-level Manager module learns to design sub-goals and a low-level Worker module recognizes the primitive actions to fulfill the sub-goal. With this compositional framework to reinforce video captioning at different levels, our approach significantly outperforms all the baseline methods on a newly introduced large-scale dataset for fine-grained video captioning. Furthermore, our non-ensemble model has already achieved the state-of-the-art results on the widely-used MSR-VTT dataset.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “, captioning an image or a video with natural language [9, 10, 39, 41, 47, 48, 42] or localizing desired objects within an image given a natural language description [31, 18, 49].”
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “Much work [38,31,15,9,40,34,33,32] has been done in language generation conditioned on visual inputs.”
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning (2018, methodology): “In Table 1, we ﬁrst list the top-3 results from the MSR-VTT Challenge 2017: v2t navigator (Jin et al., 2016), Aalto (Shetty and Laaksonen, 2016), and VideoLAB (Ramanishka et al., 2016 Bansal, 2017), Dense-Cap (Shen et al., 2017), and HRL (Wang et al., 2018).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “It is then improved by exploiting hierarchical architecture (Yu et al., 2017) and then further optimized by incorporating reinforcement learning with rewards (Wang et al., 2018a) for generating relevant and expressive narrative paragraphs.”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “While fruitful progress [44, 49, 47, 4, 48, 51, 5, 52, 50, 28, 10, 41, 2, 61, 42, 53, 59] has been made on activity detection to recognize and localize temporal segments in videos, such approaches are limited to work on pre-defined lists of simple activities, such as playing basketball, drinking wat”
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (2022, methodology): “To optimize the LM actor network, in practice, following previous work [Bahdanau et al., 2016, Rennie et al., 2017, Wang et al., 2018], in each training optimization step, we can simply approximate the expected gradient with a single sample Ws ∼ pθ: ∇θLrl(θ) ≈ −(r(W s)− r(W b)) ∑ t q̂φ(w s t )∇θ…”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “[155] proposed a hierarchical based reinforcement learning (HRL) model for describing a video.”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, background): “[51] proposed a novel hierarchical reinforcement learning framework for video captioning.”
Deep Reinforcement Learning (2018, background): “ng (NLP).Xu et al.(2015) integrate attention to image captioning. See alsoLiu et al.(2016),Lu et al.(2016),Rennie et al. (2017), andRen et al.(2017) for image captioning. SeePasunuru and Bansal(2017);Wang et al. (2018f) for video captioning.Strub et al.(2017) propose end-to-end optimization with dee”
Transferable Representation Learning in Vision-and-Language Navigation (2019, background): “A highly related class of tasks centers around generating captions for images and videos [12, 13, 37, 38, 44].”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “As a result, it has attracted increasing attention in recent years in both NLP (Venugopalan et al. 2016; Wang et al. 2018a) and computer vision communities (Krishna et al. 2017).”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “For example, [17, 18] add some auxiliary tasks like word appearance prediction; [19] uses Conditional Variational Autoencoder(CVAE) and optimize over evidence lower bound(ELBO); [20, 6, 21, 22, 23, 24] applied Reinforcement Learning algorithms on captioning, so that the models can be optimized direc”
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation (2018, methodology): “tents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance. 1 Introduction Beyond the traditional visual captioning task [36, 24, 38, 35, 31, 15] that produces one single sentence, generating long and topic-coherent stori”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, methodology): “Wang et al. (2018a) proposed a joint framework for generating reference reports and performing disease classification at the same time.”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “For instance, it is utilized for action detection in [53], for action anticipation in [14], for natural language grounding in [19] and for video description in [46].”
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (2019, background): “…contributed equally. language modalities – such as image and video captioning (Donahue et al., 2017; Fang et al., 2015; Vinyals et al., 2015; Wang et al., 2018; Yu et al., 2016), visual question answering (VQA) (Antol et al., 2015; Yang et al., 2016), and visual dialog (Das et al., 2017)…”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Dense video captioning lies at the intersection of event localization [25, 29, 33, 62, 65, 66, 84, 127] and event captioning [30,63,75,97,104].”