SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition

By Bruno Korbar, Du Tran, Lorenzo Torresani. 2019

260 citations recorded by the source corpus.

While many action recognition datasets consist of collections of brief, trimmed videos each containing a relevant action, videos in the real-world (e.g., on YouTube) exhibit very different properties: they are often several minutes long, where brief relevant clips are often interleaved with segments of extended duration containing little change. Applying densely an action recognition system to every temporal clip within such videos is prohibitively expensive. Furthermore, as we show in our experiments, this results in suboptimal recognition accuracy as informative predictions from relevant clips are outnumbered by meaningless classification outputs over long uninformative sections of the video. In this paper we introduce a lightweight ``clip-sampling'' model that can efficiently identify the most salient temporal clips within a long video. We demonstrate that the computational cost of action recognition on untrimmed videos can be dramatically reduced by invoking recognition only on these most salient clips. Furthermore, we show that this yields significant gains in recognition accuracy compared to analysis of all clips or randomly selected clips. On Sports1M, our clip sampling scheme elevates the accuracy of an already state-of-the-art action classifier by 7% and reduces by more than 15 times its computational cost.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “Further, instead of uniformly sampling 10 clips, we used SC-Sampler [4] and sampled 10 salient clips from test videos and achieved a top-1 accuracy of 82.”
Adaptive Focus for Efficient Video Recognition (2021, methodology): “In this subsection, AdaFocus is compared with several competitive baselines that focus on facilitating efficient video recognition, including MultiAgent [42], SCSampler [25], LiteEval [45], AdaFrame [44], Listen-tolook [16] and AR-Net [29].”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “More-over, unlike existing works, our framework requires neither complex RL policy gradients [61, 58, 64] nor additional modalities such as audio [14, 30] to learn dynamic policies.”
Discriminability Distillation in Group Representation Learning (2020, methodology): “Clip-based methods [50,51,13,30] use 3D convolutional neural network to jointly capture spatial-temporal features.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “Compressed video [20] and audio [12, 20] are also utilized for further improvement in clip selection.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “, R(2+1)D [51], Tiny Video Networks [44], channel-separated CNNs [50]) or selecting salient frames/clips conditioned on the input [63,62,33,20].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Given a stream of audio, IMU, and RGB video data, a model must identify the keystep being performed at each frame, as well as decide which sensor(s) to use for subsequent time-steps.”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, methodology): “Korbar et al. [48] constructed an oracle sampler and trains a clip sampler with performance close to the oracle sampler by using knowledge distillation.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “This is commonly done by training agents to find which frame to observe next [55, 52, 57, 8], ranking frames based on saliency [29], gating frames [25], skipping RNN states [4], or by using audio to select relevant frames [15, 29].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
A Multigrid Method for Efficiently Training Video Models (2019, methodology): “testing by default [25,48] and present results with other settings in Supplementary Material.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.”
Listen to Look: Action Recognition by Previewing Audio (2019, methodology): “RANDOM UNIFORM FRONT CENTER END SCSAMPLER [34] DENSE LSTM NON-RECURRENT Ours (sparse / dense)”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Representative methods typically use reinforcement learning (RL) policy gradients [61, 60] or audio [30, 17] to select relevant video frames.”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “Note that our approach is complementary to [15, 20] and the two static 3D CNNs used in these works can be replaced by adaptive 3D CNNs.”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, background): “While there is plethora of work focusing on designing robust models for video classification, limited efforts have been made on efficient video recognition [42, 36, 4, 41, 38, 5, 19, 43].”
X3D: Expanding Architectures for Efficient Video Recognition (2020, background): “K=10) from a video and spatially scales the shorter spatial side to 128γs pixels and takes a γt×112γs×112γs center crop, comparable to [36, 41, 63, 71].”