AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos

By Amlan Kar, Nishant Rai, Karan Sikka, Gaurav Sharma. 2016

159 citations recorded by the source corpus.

We propose a novel method for temporally pooling frames in a video for the task of human action recognition. The method is motivated by the observation that there are only a small number of frames which, together, contain sufficient information to discriminate an action class present in a video, from the rest. The proposed method learns to pool such discriminative and informative frames, while discarding a majority of the non-informative frames in a single temporal scan of the video. Our algorithm does so by continuously predicting the discriminative importance of each video frame and subsequently pooling them in a deep learning framework. We show the effectiveness of our proposed pooling method on standard benchmarks where it consistently improves on baseline pooling methods, with both RGB and optical flow based Convolutional networks. Further, in combination with complementary video representations, we show results that are competitive with respect to the state-of-the-art results on two challenging and publicly available benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Qiuhong Ke
3. Lu Yuan
4. Stephen Gould
5. Anoop Cherian
6. Hossein Rahmani
7. Fatih Murat Porikli
8. Zehua Sun

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Jun Liu
4. Dacheng Tao
5. Xiaogang Wang
6. G. Wang
7. Andrew Zisserman
8. Yun Raymond Fu

In Endocrinology, Diabetes and Metabolism

1. Yu Kong
2. Yun Raymond Fu
3. Jun Liu
4. Qiuhong Ke
5. Stephen Gould
6. Yichen Wei
7. Hossein Rahmani
8. Zehua Sun

Citations in the corpus

• Video Representation Learning Using Discriminative Pooling (2018, background): “HMDB-51 (accuracy over 3 splits) Method Accuracy Temporal segment networks[54] 69.4% AdaScan[19] 54.9% AdaScan + IDT + C3D[19] 66.9% ST ResNet[10] 66.4% ST ResNet + IDT[10] 70.3% ST Multiplier Network[11] 68.9% ST Multiplier Network + IDT[11] 72.2% Two-stream I3D[5] 66.4% Two-stream I3D+ (Kinetics 3”
• Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors (2021, unknown): “Superfluous frames can be discarded [31, 60].”
• Human Action Recognition and Prediction: A Survey (2018, background): “The third category of approaches uses temporal pooling (Kar et al., 2017; Girdhar et al., 2017) or aggregation to capture temporal information in a video.”
• Flow-Guided Feature Aggregation for Video Object Detection (2017, methodology): “Feature aggregation is widely used in action recognition [34, 20, 24, 47, 38, 1, 21, 41] and video description [7, 46].”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “[63] recursively predicted the discriminative importance of each frame for feature pooling.”