Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network

By Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei David Liu. 2019

184 citations recorded by the source corpus.

In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. Specifically, a gating strategy is proposed to dynamically and adaptively incorporate the global syntactic POS information into the decoder for generating each word. Experimental results on two benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed model can well exploit complementary information from multiple representations, resulting in improved performances. Moreover, the generated global POS information can well capture the global syntactic structure of the sentence, and thus be exploited to control the syntactic structure of the description. Such POS information not only boosts the video captioning performance but also improves the diversity of the generated captions. Our code is at: https://github.com/vsislab/Controllable_XGating.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “Although our model is not dedicated for video tasks, our model achieve new SOTA on MSRVD, MSRVTT, and VATEX for captioning and on MSVD-QA and TGIF-Frame for QA.”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “With the advent of the encoder-decoder framework, most of the current work is studying how to better use visual features [35, 23, 1, 36, 22, 24] and design elaborate models [34, 2, 28, 13] to generate sentences directly.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “It is worth noting that different from the work [54], where POS (Part-of-Speech) tag of each word is predicted to guide the captioning process, we only focus on the components that convey the most visual information to target action and guide caption generation.”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, background): “Input text (e.g., ASR captions) are tokenized and represented as a sequence of WordPiece (Wu et al., 2016) tokens.”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, unknown): “5 POS+CG [51] IncepResnetV2 OpticalFlow 52.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “5 POS+CG [36] 2019 InceptionResnetV2 OpticalFlow - 52.”
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “We follow the standard data split scheme in previous work [30, 39, 49]: 6513 video clips in training set, 497 in validation, and 2990 in testing.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “We adopt the set of features commonly used by recent state-of-the-art methods [59,1,9], which are Inception-ResNet v2 and C3D for visual and motion modalities, respectively.”