Generalized Rank Pooling for Activity Recognition

By Anoop Cherian, Basura Fernando, Mehrtash Harandi, Stephen Gould. 2017

87 citations recorded by the source corpus.

Most popular deep models for action recognition split video sequences into short sub-sequences consisting of a few frames, frame-based features are then pooled for recognizing the activity. Usually, this pooling step discards the temporal order of the frames, which could otherwise be used for better recognition. Towards this end, we propose a novel pooling method, generalized rank pooling (GRP), that takes as input, features from the intermediate layers of a CNN that is trained on tiny sub-sequences, and produces as output the parameters of a subspace which (i) provides a low-rank approximation to the features and (ii) preserves their temporal order. We propose to use these parameters as a compact representation for the video sequence, which is then used in a classification setup. We formulate an objective for computing this subspace as a Riemannian optimization problem on the Grassmann manifold, and propose an efficient conjugate gradient scheme for solving it. Experiments on several activity recognition datasets show that our scheme leads to state-of-the-art performance.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Representation Learning Using Discriminative Pooling (2018, background): “There have been other recent works that use parameters of machine learning algorithms for the purpose of pooling, such as rank pooling [14], generalized rank pooling [6], dynamic images [2] and dynamic flow [52].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Generalized rank pooling [55] is introduced to improve the original method via a quadratic ranking function which jointly provides a low-rank approximation to the input data and preserves their temporal order in a subspace.”
Building Deep Networks on Grassmann Manifolds (2016, background): “2014b) and activity recognition (Cherian et al. 2017), one can typically model the set of images of a single person well by a low dimensional subspace, and then compare such subspaces directly.”
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling (2019, methodology): “A rich set of methods have explicitly exploited it for video recognition, including ranking pooling [9, 4], and RNN-based methods [7, 34].”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “We have moved from manually designed features [1], [2], [3], [4], [5] to learned convolutional neural network (CNN) features [6], [7], [8], [9]; from encoding appearance information to encoding motion information [10], [11], [12], [13]; and from learning local features to learning global video featu”