Less Is More: Picking Informative Frames for Video Captioning

By Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang. 2018

209 citations recorded by the source corpus.

In video captioning task, the best practice has been achieved by attention-based models which associate salient visual components with sentences in the video. However, existing study follows a common procedure which includes a frame-level appearance modeling and motion modeling on equal interval frame sampling, which may bring about redundant visual information, sensitivity to content noise and unnecessary computation cost. We propose a plug-and-play PickNet to perform informative frame picking in video captioning. Based on a standard Encoder-Decoder framework, we develop a reinforcement-learning-based procedure to train the network sequentially, where the reward of each frame picking action is designed by maximizing visual diversity and minimizing textual discrepancy. If the candidate is rewarded, it will be selected and the corresponding latent representation of Encoder-Decoder will be updated for future trials. This procedure goes on until the end of the video sequence. Consequently, a compact frame subset can be selected to represent the visual information and perform video captioning without performance degradation. Experiment results shows that our model can use 6-8 frames to achieve competitive performance across popular benchmarks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Cordelia Schmid
4. Xiaogang Wang
5. Jitendra Malik
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Zhe Gan

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Zhengyuan Yang
7. Xiaowei Hu
8. Ce Liu

Citations in the corpus

• Attention on Attention for Image Captioning (2019, neutral): “[42, 27, 47, 2, 7, 16, 15] and achieved impressive results.”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[12] Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang.”
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, background): “Prior works [13] also studied frame selection schemes to capture informative visual inputs.”
• Recurrent Fusion Network for Image Captioning (2018, background): “Captioning [1,2,3,4,5,6,7], a task to describe images/videos with natural sentences automatically, has been an active research topic in computer vision and machine learning.”
• Efficient Video Classification Using Fewer Frames (2019, background): “In some very recent work on video classification [10] and video captioning [6] the authors use a reinforcement learning agent to select which frames to process.”
• Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “We first compare our approach against earlier methods, including RecNet [40], which adds one reconstructor on top of the traditional encoder-decoder framework to reconstruct the visual features from the generated caption, and PickNet [6] which dynamically attends to frames by maximizing a picking pol”
• Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “Nowadays, benefit from the success of CNN and RNN, the sequence learning methods [42, 53, 27, 28, 8, 45, 49] are widely used to describe video content with flexible syntactic structure.”
• SibNet: Sibling Convolutional Encoder for Video Captioning (2018, background): “[43] proposed to generate natural language captions based on a few informative frames picked by their method rather than all the frames of the video.”
• Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “7 PickNet∗ [6] 2018 ResNet-152 - 52.”
• Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “[28] 2018 RNN/LSTM, RNN/GRU Video, Text MSVD, MSR-VTT BLEU, ROUGEL , METEOR, CIDEr.”
• Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “Specifically, we choose i) fundamental methods including SA [60], M3 [56], MA-LSTM [59], VideoLab [40], v2t navigator [23], ii) latest state-of-the-art methods including PickNet [6], RecNet local [55], OA-BTG [65], MARN [38], GRU-EVE [1], and POS-CG 3 .”
• Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “Most existing methods of video captioning [48, 41, 13, 30, 4, 12, 8, 1] mainly focus on investigating various visual perception models by exploiting informative semantics without considering learning of syntax representation for generating sentences.”
• A Better Variant of Self-Critical Sequence Training (2020, background): “…Chen et al., 2019; Yang et al., 2019; Zhao et al., 2018), but also has been applied to many other tasks, like video cap-tioning(Li et al., 2018; Chen et al., 2018; Li and Gong, 2019), reading comprehension(Hu et al., 2017), summarization(Celikyilmaz et al., 2018; Paulus et al., 2017; Wang et…”
• Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “We compare performance of our approach on this dataset with the latest published models such as Alto [41], RUC-UVA [14], TDDF [59], PickNet [12], M 3 -VC [53] and RecNet local [51].”