Video Summarization With Attention-Based Encoder–Decoder Networks

By Zhong Ji, Kailin Xiong, Yanwei Pang, Xuelong Li. IEEE transactions on circuits and systems for video technology (Print), 2017

373 citations recorded by the source corpus.

This paper addresses the problem of supervised video summarization by formulating it as a sequence-to-sequence learning problem, where the input is a sequence of original video frames, and the output is a keyshot sequence. Our key idea is to learn a deep summarization network with attention mechanism to mimic the way of selecting the keyshots of human. To this end, we propose a novel video summarization framework named attentive encoder–decoder networks for video summarization (AVS), in which the encoder uses a bidirectional long short-term memory (BiLSTM) to encode the contextual information among the input video frames. As for the decoder, two attention-based LSTM networks are explored by using additive and multiplicative objective functions, respectively. Extensive experiments are conducted on two video summarization benchmark datasets, i.e., SumMe and TVSum. The results demonstrate the superiority of the proposed AVS-based approaches against the state-of-the-art approaches, with remarkable improvements on both datasets.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/88a8baa1be5292e62622f1cb8e627fbf759bf741

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Alexandros I. Metsai
2. Eleni Adamantidou
3. Zhong Ji
4. Yanwei Pang
5. Yuxiao Zhao
6. Vasileios Mezaris
7. Jungong Han
8. Xi Li

In Signal Processing

1. Zhong Ji
2. Yanwei Pang
3. Jungong Han
4. Xi Li
5. Ioannis Patras

Citations in the corpus

• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, neutral): “Many authors [1, 4, 6, 12, 17, 18, 26, 27, 23, 28, 33] use a spatial softmax to compute attention weights.”
• Deep Attentive Video Summarization With Distribution Consistency Learning (2020, methodology): “Video summarization, also called video abstraction or video skim, has been an active research topic for more than two decades [3], [14], [15].”
• Video Summarization Using Deep Neural Networks: A Survey (2021, methodology): “[28] formulates video summarization as a seq2seq learning problem and proposes an LSTM-based encoder-decoder network with an intermediate attention layer.”