Video Summarization with Long Short-Term Memory
By Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman. 2016
783 citations recorded by the source corpus.
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots. Casting the problem as a structured prediction problem on sequential data, our main idea is to use Long Short-Term Memory (LSTM), a special type of recurrent neural networks to model the variable-range dependencies entailed in the task of video summarization. Our learning models attain the state-of-the-art results on two benchmark video datasets. Detailed analysis justifies the design of the models. In particular, we show that it is crucial to take into consideration the sequential structures in videos and model them. Besides advances in modeling techniques, we introduce techniques to address the need of a large number of annotated data for training complex learning models. There, our main idea is to exploit the existence of auxiliary annotated video datasets, albeit heterogeneous in visual styles and contents. Specifically, we show domain adaptation techniques can improve summarization by reducing the discrepancies in statistical properties across those datasets.
Subfields: Computer Vision and Pattern Recognition, Model Architectures, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Yi Yang
- Ming-Hsuan Yang
- Xiaogang Wang
- Andrew Zisserman
In Model Architectures
- Marcus Rohrbach
- Trevor Darrell
- Anna Rohrbach
- Michael Lewis
- Lorenzo Torresani
- Luowei Zhou
- Sinisa Todorovic
- Xi Li
In Summarization
Citations in the corpus
- Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “We use long short-term memory (LSTM) for sequential prediction due to its state-of-the-art performance in sequence modeling (Xu et al. 2016; Zhang et al. 2016).”
- End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “Video summarization [41] is one way of doing this, but it loses the language components of the video, which are particularly important in instructional videos.”
- COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, neutral): “, action temporal localization [45, 50], video summarization [19, 30, 49] and video caption [27, 47, 53], etc) have been developed recently.”
- Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Works motivated by other application scenarios usually explore hierarchical modeling in one modality [24, 43, 45].”
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “Previous studies [5, 46, 49, 58] tend to settle each task separately on specialized downstream datasets.”
- A Dataset for Telling the Stories of Social Media Videos (2018, background): “Other related work includes visual summarization of Flickr photo albums (Sigurds-son et al., 2016a) or videos (De Avila et al., 2011; Zhang et al., 2016) where the idea is to pick the key images or frames that summarize the visual content.”
- Video Summarization Using Deep Neural Networks: A Survey (2021, methodology): “The evaluation approach and benchmark datasets of [19] and [20] were jointly used to evaluate the summarization performance in [21].”
- SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, background): “For example, video summarization [18, 19, 29, 37, 56, 57, 58] and the automatic production of sport highlights [30, 31] entail creating a much shorter version of the original video by concatenating a small set of snippets corresponding to the most informative or exciting moments.”
- Unsupervised Video Summarization with Adversarial LSTM Networks (2017, methodology): “We follow the steps in [43] to convert framelevel scores to key frames and key shot summaries, and vice versa in all datasets.”
- Summarization of Egocentric Videos: A Comprehensive Survey (2017, methodology): “The LSTM-based summarization method presented in [24] is evaluated using the f -measure, precision, and recall of the selected segments against the annotation, with both TPV and FPV videos.”
- CLIP-It! Language-Guided Video Summarization (2021, methodology): “The ground truth summary shows the scores computed by averaging the annotations from all users as in [43] and the keyframes that received high scores.”
- Weakly supervised classification of rare aortic valve malformations using unlabeled cardiac MRI sequences (2018, methodology): “The final hybrid CNN-LSTM architecture aligns with 231 recent methods for state-of-the-art video classification [37,38] and 3D medical 232 imaging [39].”
- Deep Attentive Video Summarization With Distribution Consistency Learning (2020, methodology): “loss, especially mse, is the most popular loss employed in video summarization [2], [3], [10], [29], and it minimizes the”
- A Practical Online Framework for Extracting Running Video Summaries under a Fixed Memory Budget (2021, methodology): “To assess the quality of a streaming summarization technique, we employ a time dependent F-measure based evaluation method [10, 11, 27, 12, 32, 35, 22].”
- Dense Procedure Captioning in Narrated Instructional Videos (2019, methodology): “We compare our model with several baseline methods: (1) SCNN-prop (Shou et al., 2016) ; (2) vsLSTM is an LSTM based video summarization model (Zhang et al., 2016); (3) Proc-Nets (Zhou et al., 2018a) which is the previous SOTA method.”
- Machine Remaining Useful Life Prediction via an Attention-Based Deep Learning Approach (2021, background): “Owing to its unique property, the LSTM network has achieved great successes in the analysis of time-series data, such as occupancy estimation [30], video analysis [31], and nature language processing [32].”
- Video Summarization With Attention-Based Encoder–Decoder Networks (2017, methodology): “To the best of our knowledge, the only existing RNN-based video summarization approaches are [17] and [18].”