Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

By Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu. 2015

581 citations recorded by the source corpus.

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal-and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Work in this area encompasses image captioning (Lin et al., 2014), video captioning (Yu et al., 2016), visual QA (Antol et al., 2015), and more conversationally, visual dialogue (Das et al., 2017).”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “The output procedure segments of ProcNets can be applied for other tasks, including full agent-based procedure learning (Yu and Siskind 2013) or smaller-scale video description generation (Yu et al. 2016; Krishna et al. 2017).”
Dense-Captioning Events in Videos (2017, methodology): “A recent paper has attempted to describe videos with multiple sentences [51].”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “For example, LSTM-YT [32], S2YT [31], TempoAttn [36], H-RNN [41] and DEM [20].”
Localizing Moments in Video with Natural Language (2017, background): “Contemporary models integrate deep video representations with recurrent language models [25, 31, 47, 48, 56].”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “According to the BLEU metric, we also outperform other approaches using more complex decoder schemes such as spatial and temporal attention decoder (Yu et al., 2015) or a hierarchical RNN decoder (Pan et al., 2015) Our approach is on par with Yu et al. (2015), without the need of using a…”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “[43] introduce a hierarchical RNN decoder , speciﬁcally Gated Recurrent Unit (GRU) [7], into the video captioning system.”
Representation of Linguistic Form and Function in Recurrent Neural Networks (2016, background): “RNNs are also important components of systems integrating vision and language—for example, image (Karpathy and Fei-Fei 2015) and video captioning (Yu et al. 2015).”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “There are also setups that consider static language and perception, for example image captioning (Lin et al., 2014), video captioning (Yu et al., 2016), visual QA (Antol et al., 2015) and visual dialogue (Das et al., 2017; Shuster et al., 2018; Mostafazadeh et al., 2017).”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “In vision, language is also used in captioning where the task is to generate a descriptive caption of an image or a video [10], [25], [26], [27].”
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (2016, background): “Later this soft attention has been extended as temporal attention over video frames [33, 35] for video captioning.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “, captioning an image or a video with natural language [9, 10, 39, 41, 47, 48, 42] or localizing desired objects within an image given a natural language description [31, 18, 49].”
Semantic Compositional Networks for Visual Captioning (2016, background): “Most of these approaches learn a probabilistic model of the caption, conditioned on an image or a video [29, 47, 13, 20, 48, 52, 10, 46, 32, 56].”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “The pixel-level approach trains the model with raw video as input directly (Torabi et al., 2016; Kiros et al., 2014; Yu et al., 2016a; Kaufman et al., 2017; Yu et al., 2017, 2018).”
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “Much work [38,31,15,9,40,34,33,32] has been done in language generation conditioned on visual inputs.”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, background): “One of the major challenges in video understanding [7, 21, 19, 23, 24, 20, 5] is to localize and classify human actions in long, untrimmed videos, of which usually the requirement is to predict per-frame semantic labels.”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Works motivated by other application scenarios usually explore hierarchical modeling in one modality [24, 43, 45].”