Reconstruction Network for Video Captioning

By Bairui Wang, Lin Ma, Wei Zhang, Wei David Liu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

353 citations recorded by the source corpus.

In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) with a novel encoder-decoder-reconstructor architecture, which leverages both the forward (video to sentence) and backward (sentence to video) flows for video captioning. Specifically, the encoder-decoder makes use of the forward flow to produce the sentence description based on the encoded video semantic features. Two types of reconstructors are customized to employ the backward flow and reproduce the video features based on the hidden state sequence generated by the decoder. The generation loss yielded by the encoder-decoder and the reconstruction loss introduced by the reconstructor are jointly drawn into training the proposed RecNet in an end-to-end fashion. Experimental results on benchmark datasets demonstrate that the proposed reconstructor can boost the encoder-decoder models and leads to significant gains in video caption accuracy.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ba7405516e1408f0ee6e0d0a8c6d511ce33c0551

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences (2018, background): “(Shen et al. 2018; Wang et al. 2018; Venugopalan et al. 2015) and the recent GIF-Text cross-modal retrieval (Song and Soleymani 2018).”
Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning (2019, neutral): “…CIDEr Our OA-BTG 56.9 36.2 90.6 SeqVLAD [6] 51.0 35.2 86.0 LSTM-GAN [23] 42.9 30.4 - MS-RNN [31] 53.3 33.8 74.8 MCNN+MCF [32] 46.5 33.7 75.5 RecNet [33] 52.3 34.1 80.3 TSA-ED [34] 51.7 34.0 74.9 aLSTMs [35] 50.8 33.3 74.8 STAT [36] 51.1 32.7 67.5 MA-LSTM [16] 52.3 33.6 70.4 DMRM [2] 51.1 33.6 74.8…”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Recent event captioning methods [37,47,60,9,26] mostly adopt an encoder-decoder structure, where the encoder aggregates video features and the decoder (usually LSTM [24] or GRU [13]) generates sentences based on the aggregation results.”
Less Is More: Picking Informative Frames for Video Captioning (2018, methodology): “With the development of neural networks and deep learning, modern captioning systems are based on CNN, RNN and the encoder-decoder architecture [35, 36].”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “With highly rich visual codes, a relatively simple Gated Recurrent Unit (GRU) network is proposed for language modeling, comprising two layers, that already results in on-par or better performance compared to the existing sophisticated models [52, 54, 34, 18] on multiple evaluation metrics.”
VLP: A Survey on Vision-language Pre-training (2022, neutral): “Visual Captioning (VC) [85, 40, 86].”
Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “…- 51.7 HRNE [29] 46.7 33.9 - - h-RNN [48] 49.9 32.6 - 65.8 BAE [4] 42.5 32.4 - 63.5 TSA [30] 52.8 33.5 - 74.0 aLSTMs [13] 50.8 33.3 61.1 74.8 SCN [12] 51.1 33.5 - 77.7 M 3 [40] 52.8 33.1 - - RecNet [39] 52.3 the improvement of our method under B@4 is not as remarkable as other evaluation metrics.”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “Video understanding, especially action recognition, has been extensively studied for decades, where video representation learning serves as the fundamental problem of other video-related tasks, such as complex action recognition [23], action temporal localization [8, 41, 42], video caption [49, 52],”
Recurrent Fusion Network for Image Captioning (2018, background): “Captioning [1,2,3,4,5,6,7], a task to describe images/videos with natural sentences automatically, has been an active research topic in computer vision and machine learning.”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “Nowadays, benefit from the success of CNN and RNN, the sequence learning methods [42, 53, 27, 28, 8, 45, 49] are widely used to describe video content with flexible syntactic structure.”
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “We ﬁrst compare our approach against earlier methods, including RecNet [40], which adds one reconstructor on top of the traditional encoder-decoder framework to reconstruct the visual features from the generated caption, and PickNet [6] which dynamically attends to frames by maximizing a picking…”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “It is then improved by exploiting hierarchical architecture (Yu et al., 2017) and then further optimized by incorporating reinforcement learning with rewards (Wang et al., 2018a) for generating relevant and expressive narrative paragraphs.”
Temporally Grounding Natural Sentence in Video (2018, background): “…retrieval (Xu et al., 2015b), video attractiveness prediction (Chen et al., 2018, 2016), and video captioning (Pasunuru and Bansal, 2017; Wang et al., 2018a,b), NSGV needs to model not only the characteristics of sentence and video but also the fine-grained interactions between the two…”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “RecNet [55] employs backward ﬂow to reproduce the video features while generating descriptions.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, background): “which is a promising and challenging task [20, 37, 39, 52].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “[149] proposed a reconstruction network for video description using an encoderdecoder reconstructor architecture, which utilizes both forward flow (from video to sentence) and backward flow (from sentence to video).”
Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (2019, background): “Learning powerful spatio-temporal representations is the most fundamental deep learning problem for many video understanding tasks such as action recognition [4, 17, 26], action proposal and localization [5, 33, 34], video captioning [40, 42], etc.”
Egocentric Video-Language Pretraining (2022, background): “With the recent interest boom in computer vision and natural language processing, Video-Language Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-text retrieval [1, 2”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Dense video captioning lies at the intersection of event localization [25, 29, 33, 62, 65, 66, 84, 127] and event captioning [30,63,75,97,104].”