Watching a Small Portion could be as Good as Watching All: Towards Efficient Video Classification

By Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Clarence Yan, Jianjun Ge, Yi Yang. 2018

85 citations recorded by the source corpus.

We aim to significantly reduce the computational cost for classification of temporally untrimmed videos while retaining similar accuracy. Existing video classification methods sample frames with a predefined frequency over entire video. Differently, we propose an end-to-end deep reinforcement approach which enables an agent to classify videos by watching a very small portion of frames like what we do. We make two main contributions. First, information is not equally distributed in video frames along time. An agent needs to watch more carefully when a clip is informative and skip the frames if they are redundant or irrelevant. The proposed approach enables the agent to adapt sampling rate to video content and skip most of the frames without the loss of information. Second, in order to have a confident decision, the number of frames that should be watched by an agent varies greatly from one video to another. We incorporate an adaptive stop network to measure confidence score and generate timely trigger to stop the agent watching videos, which improves efficiency without loss of accuracy. Our approach reduces the computational cost significantly for the large-scale YouTube-8M dataset, while the accuracy remains the same.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “[56], [57] and texts [58], [59], [60] (Sec.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, methodology): “Representative methods typically use Reinforcement Learning (RL) where an agent [62,14,63] or multiple agents [59] are trained with policy gradient methods to select relevant video frames, without deciding frame resolution as in our approach.”
Reducto: On-Camera Filtering for Resource-Efficient Real-Time Video Analytics (2020, background): “The idea of filtering frames based on their features is widely seen in the CV community [18, 29, 44, 60, 70, 73, 74].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, methodology): “Comparisons of AdaFrame with FrameGlimpse [36], FastForward [3], and alternative frame selection methods based on heuristics.”
Efficient Video Classification Using Fewer Frames (2019, background): “In some very recent work on video classification [10] and video captioning [6] the authors use a reinforcement learning agent to select which frames to process.”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, unknown): “While they are conceptually similar to our approach, which also aims to skip redundant frames, our framework is fully differentiable, and thus is easier to train than policy search methods [4, 41].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Note that FrameGlimpse [61], FastForward [11] and AdaFrame [60] have less computation as they require access to future frames unlike LiteEval and AdaMML that makes decision based on the current time stamp only.”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “Backbone Pre-trained top-1 mAP IDT [ sults of these methods are directly cited from Adaframe [51] whose authors reproduce FrameGlimpse [53] and Fast-Forward [9].”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “This is commonly done by training agents to find which frame to observe next [55, 52, 57, 8], ranking frames based on saliency [29], gating frames [25], skipping RNN states [4], or by using audio to select relevant frames [15, 29].”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, background): “Meanwhile, several works [29], [30], [31], [5], [32], [33] attempted to utilize long-term temporal information for action recognition.”
Adaptive Focus for Efficient Video Recognition (2021, background): “, terminating the computation before “watching” the full sequence [10, 43]; (2) conditional computing, e.”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, unknown): “This is typically achieved by means of temporal models that “skip” segments by leveraging past observations to predict which future frames to consider next [55, 10, 53].”
Listen to Look: Action Recognition by Previewing Audio (2019, methodology): “Several recent methods [52, 14, 71, 69] propose reinforcement learning (RL) approaches for video frame selection.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Similar in spirit, dynamic methods for efﬁcient video recognition adaptively select salient frames/clips [64, 61, 30, 9, 58, 23], utilize audio [14], reduce feature redundancy [38], or select frame resolutions [60, 34].”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “…and non-differentiable, we rely on a Gumbel Softmax sampling approach (Jang et al., 2016) to learn the policy jointly with the network parameters through standard back-propagation, without resorting to complex reinforcement learning as in Wu et al. (2019b); Fan et al. (2018); Yeung et al. (2016).”