Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning

By Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, Yueting Zhuang. 2015

397 citations recorded by the source corpus.

Recently, deep learning approach, especially deep Convolutional Neural Networks (ConvNets), have achieved overwhelming accuracy with fast processing speed for image classification. Incorporating temporal structure with deep ConvNets for video representation becomes a fundamental problem for video content analysis. In this paper, we propose a new approach, namely Hierarchical Recurrent Neural Encoder (HRNE), to exploit temporal information of videos. Compared to recent video representation inference approaches, this paper makes the following three contributions. First, our HRNE is able to efficiently exploit video temporal structure in a longer range by reducing the length of input information flow, and compositing multiple consecutive inputs at a higher level. Second, computation operations are significantly lessened while attaining more non-linearity. Third, HRNE is able to uncover temporal tran-sitions between frame chunks with different granularities, i.e. it can model the temporal transitions between frames as well as the transitions between segments. We apply the new method to video captioning where temporal information plays a crucial role. Experiments demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “In vision, language is also used in captioning where the task is to generate a descriptive caption of an image or a video [10], [25], [26], [27].”
Reinforced Video Captioning with Entailment Rewards (2017, methodology): “…sequence-to-sequence models for video captioning, using attention, hierarchical RNNs, 3D-CNN video features, joint embedding spaces, language fusion, etc., but using word-level cross entropy loss training (Venugopalan et al., 2015a; Yao et al., 2015; Pan et al., 2016a,b; Venugopalan et al., 2016).”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, methodology): “We differ from previous works [19, 25] in two aspects (components in red color): layer-wise reconstruction through decoders, and matching at both global and local levels.”
Visual to Text: Survey of Image and Video Captioning (2019, methodology): “S2VT [102], HRNE [107] and LSTM-BA [108]; (2) the spatialtemporal CNN based encoding methods including SA [103], h-RNN [106], LSTM-E [107] and LSTM-TSA [109].”
Motion Guided Spatial Attention for Video Captioning (2019, methodology): “Our contributions of this work are as follows: (1) We present a novel video captioning framework named Motion Guided Spatial Attention (MGSA), which utilizes optical ﬂow to guide spatial attention.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Recent works [73,37,9,44,11,43,41] suggest that further improvements can be obtained by additionally leveraging motion and audio representations.”
Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “Many deep learning methods [11, 38, 37, 29] usually build an encoder to compress the input video into a feature representation and a decoder to generate descriptions given the video feature.”
Watching a Small Portion could be as Good as Watching All: Towards Efficient Video Classification (2018, methodology): “Video classiﬁcation model generates representations which are usually beneﬁcial to other video analysis tasks, i.e. , video captioning [Pan et al. , 2016] and video question answering [Zhu et al. , 2017].”
Multi-Task Video Captioning with Video and Entailment Generation (2017, background): “8In their updated supplementary at https: //www.microsoft.com/en-us/research/wp-content/ uploads/2016/10/cvpr16.supplementary.pdf 9 http://ms-multimedia-challenge.com/leaderboard Pan et al., 2016a; Yao et al., 2015).10”
Few-Shot Object Recognition from Machine-Labeled Web Images (2016, background): “LSTM has resurged due to the success of sequence to sequence modeling [29] on machine translation [3], image captioning [34, 17, 37], video classification [38], video captioning [31, 25], etc.”
Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models (2018, neutral): “video interpretation [21, 27], language modeling [25, 28–30], speech recognition [31, 32], machine translation [33] and text classification [34, 35], etc.”
Uncovering the Temporal Context for Video Question Answering (2017, background): “In the area of video analysis, a small number of recent systems have been proposed for video captioning (Rohrbach et al. 2013; Venugopalan et al. 2015; Yao et al. 2015; Pan et al. 2016).”
Learning Multimodal Attention LSTM Networks for Video Captioning (2017, methodology): “Previous research [13, 14, 24, 30, 33] simply concatenates different features into one single video representation, while neglecting the intrinsic modality nature.”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “The ﬁgure also reports the captions for GRU-MP-(CI) and GRU-EVE hft -(CI) to show the difference resulting from hi-Table 29.2 LSTM-YT [50] 29.07 TA [57] 29.0 p-RNN [58] 31.1 HRNE [31] 33.1 BGRCN [5] 31.70 MAA [16] 31.80 RMA [22] 31.90 LSTM-E [32] 29.5 M 3 -inv3 [53] 32.18 GRU-EVE hft -(IRV2) 33.7 1.”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, methodology): “The idea of such a hierarchy is generic and has been explored by several works [21, 24, 22] in the context of video-text learning.”