AR-Net: Adaptive Frame Resolution for Efficient Action Recognition

By Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, Rogério Schmidt Feris. 2020

177 citations recorded by the source corpus.

Action recognition is an open and challenging problem in computer vision. While current state-of-the-art models offer excellent recognition results, their computational expense limits their impact for many real-world applications. In this paper, we propose a novel approach, called AR-Net (Adaptive Resolution Network), that selects on-the-fly the optimal resolution for each frame conditioned on the input for efficient action recognition in long untrimmed videos. Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency. We efficiently train the policy network jointly with the recognition model using standard back-propagation. Extensive experiments on several challenging action recognition benchmark datasets well demonstrate the efficacy of our proposed approach over state-of-the-art methods. The project page can be found at this https URL

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “In particular, on ActivityNet and mini-kinetics, we outperform AR-Net [33], the leading method among competitors, with 1.3× and 2.1× less GFLOPs, respectively.”
Adaptive Focus for Efficient Video Recognition (2021, methodology): “To address this issue, a number of recent works propose to reduce the inherent temporal redundancy in video recognition [26, 29, 45, 42, 16, 44, 25].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Thus, most of the existing methods have adopted gated RNN architectures, such as Long-Short Term Memory (LSTM) [67], [68], [69], [70], to”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Motivated by this, we leverage the idea of dynamic inference [39, 37, 38, 61, 57], and adopt a policy network (referred to as multi-head interpreter) to decide which patches are uninformative and then discard them.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Given a stream of audio, IMU, and RGB video data, a model must identify the keystep being performed at each frame, as well as decide which sensor(s) to use for subsequent time-steps.”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “It is worth pointing out that our method is orthogonal to adaptive methods on selecting salient frames/clips, input resolutions and salient spatial regions [15, 22, 39, 44, 46], and thus could be complementary to them.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “Mini-Kinetics-200 (assembled by (Meng et al., 2020)) is a subset of full Kinetics dataset (Carreira & Zisserman, 2017) containing 121k videos for training and 10k videos for testing across 200 action classes.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, unknown): “3.3.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “As shown in Table 4, on Something V1, Jester and Mini-Kinetics, we achieve a better accuracy-efﬁciency trade-off than AR-Net while using 40% less parameters.”