Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video

By Yu-Chuan Su, Kristen Grauman. 2016

31 citations recorded by the source corpus.

Current approaches for activity recognition often ignore constraints on computational resources: (1) they rely on extensive feature computation to obtain rich descriptors on all frames, and (2) they assume batch-mode access to the entire test video at once. We propose a new active approach to activity recognition that prioritizes “what to compute when” in order to make timely predictions. The main idea is to learn a policy that dynamically schedules the sequence of features to compute on selected frames of a given test video. In contrast to traditional static feature selection, our approach continually re-prioritizes computation based on the accumulated history of observations and accounts for the transience of those observations in ongoing video. We develop variants to handle both the batch and streaming settings. On two challenging datasets, our method provides significantly better accuracy than alternative techniques for a wide range of computational budgets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Existing arts [168], [169], [174] typically learn to predict the location that the network should jump to at each time step.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “Reinforcement learning has also been applied for dynamic feature prioritization in images [26] and video [45, 56], actively deciding which frames or image regions to visit next.”
Next-active-object prediction from egocentric videos (2017, background): “McCandless and Grauman [29] proposed to learn the spatio-temporal partitions which were most discriminative for a set of egocentric activities.”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, background): “nducted for video analysis tasks, discovering minimally needed evidence for event recognition [9] and training agents that decide which frames need to be observed in order to localize actions in time [50, 46]. Motivated by the advantages of training recurrent models on shorter subsequences, efforts ”
X3D: Expanding Architectures for Efficient Video Recognition (2020, result): “[1,36,52,73,74,78], which we think can be complementary to our approach.”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “There are various efforts on temporal action detection or finding action segments in untrimmed videos like [1, 23, 33, 28].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, methodology): “Only a few methods consider efficient video classification [38, 31, 40, 20, 10].”
SST: Single-Stream Temporal Action Proposals (2017, background): “A number of existing methods approach the problem of temporal action localization [19, 22, 26, 27, 31, 32, 35, 40, 24, 30, 38].”
Listen to Look: Action Recognition by Previewing Audio (2019, methodology): “Several recent methods [52, 14, 71, 69] propose reinforcement learning (RL) approaches for video frame selection.”