AdaFrame: Adaptive Frame Selection for Fast Video Recognition

By Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, Larry S. Davis. 2018

256 citations recorded by the source corpus.

We present AdaFrame, a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame contains a Long Short-Term Memory network augmented with a global memory that provides context information for searching which frames to use over time. Trained with policy gradient methods, AdaFrame generates a prediction, determines which frame to observe next, and computes the utility, i.e., expected future rewards, of seeing more frames at each time step. At testing time, AdaFrame exploits predicted utilities to achieve adaptive lookahead inference such that the overall computational costs are reduced without incurring a decrease in accuracy. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet. AdaFrame matches the performance of using all frames with only 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We further qualitatively demonstrate learned frame usage can indicate the difficulty of making classification decisions; easier samples need fewer frames while harder ones require more, both at instance-level within the same class and at class-level among different categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, unknown): “While there is plethora of work focusing on designing robust models for video classification, limited efforts have been made on efficient video recognition [42, 36, 4, 41, 38, 5, 19, 43].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Moreover, our framework is fully differentiable, and thus is easier to train than complex RL policy gradients [61, 60, 57].”
X3D: Expanding Architectures for Efficient Video Recognition (2020, result): “[1,36,52,73,74,78], which we think can be complementary to our approach.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “Combining Ada3D with AdaFrame [43] offers computational savings for video-level aggregation.”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, methodology): “[52] proposed the AdaFrame framework to determine which frames based on the LSTM [53] module, and the whole framework was trained by policy gradient methods.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “AdaFrame and LiteEval both use MobileNet-V2 as the policy network and ResNet-101 as the recognition network.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “spatial attention [78], [79], [80], [81], temporal attention [82], [83], and both spatial and temporal attention [84], [85], have”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “To illustrate the efﬁciency of our framework, we compute the ﬂoating point operations (FLOPS), which is a hardware-independent metric and widely used in the ﬁeld of efﬁcient action recognition 1 (Wu et al., 2019b; Gao et al., 2020; Meng et al., 2020; Fan et al., 2019).”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “We quote the reported results on MultiAgent [59], AdaFrame[62] and ListenToLook[20] (here “(IA|R)” and “(MN|R)” are short for “(Image-Audio|ResNet-101)” and “(MobileNetV2|ResNet-101)” mentioned in [20]).”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Motivated by the need of efﬁciency, existing research efforts mainly focus on either designing compact models [41, 49, 11] or sampling of salient frames for efﬁcient recognition [61, 58, 34].”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “By integrating SGS as an additional layer within current 3D CNNs, which use static temporal feature resolutions, we can convert them into much more efﬁcient 3D CNNs with adaptive temporal feature resolutions (ATFR) .”
Revisiting the “Video” in Video-Language Understanding (2022, background): “…note that aspects of our technical approach draw inspiration from efﬁcient image-centric video modeling literature, which aim to improve efﬁciency and for tasks like action recognition [53] and localization [57] by learning how to selectively process a sparse number of frames from the input video.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video…”
Listen to Look: Action Recognition by Previewing Audio (2019, methodology): “Without using guidance from strong human supervision, they ease the learning process by restricting the agent to a rigid action space [14], guiding the selection process of the agent with a global memory module [71], or using multiple agents to collaboratively perform frame selection [69].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “sults of these methods are directly cited from Adaframe [51] whose authors reproduce FrameGlimpse [53] and FastForward [9].”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, background): “Our network is also relevant to dynamic network designs [33, 36, 3, 40, 24].”