Rank Pooling for Action Recognition

By Basura Fernando, Efstratios Gavves, José Oramas, Amir Ghodrati, Tinne Tuytelaars. 2015

309 citations recorded by the source corpus.

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g., how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. G. Wang
3. Wanqing Li
4. Pichao Wang
5. Alex ChiChung Kot
6. Hossein Rahmani
7. Amir Shahroudy
8. Yi Yang

In Biomedical Engineering

1. Wanqing Li
2. Pichao Wang
3. Jun Liu
4. G. Wang
5. Zhimin Gao
6. Philip O. Ogunbona
7. Hossein Rahmani
8. Yonghong Hou

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yi Yang
3. G. Wang
4. Xiaogang Wang
5. Dacheng Tao
6. Jun Liu
7. Ming-Hsuan Yang
8. Trevor Darrell

Citations in the corpus

• Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, background): “HUMAN action recognition is a fast developing research area due to its wide applications in intelligent surveillance, human-computer interaction, robotics, and so on [1], [2].”
• Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, methodology): “In the spirits of Eulerian motion and rank pooling methods [5, 1], we propose to encode SFM sequence into SFAM.”
• Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, methodology): “Concurrent with our work, another state-of-the-art method applies a strategy called ranked pool [13] that generates a fast video-level descriptor, namely, the dynamic images [3].”
• Spatially and Temporally Structured Global to Local Aggregation of Dynamic Depth Information for Action Recognition (2018, background): “[30], [34] is restricted to exploit long term dynamics.”
• Discriminative Hierarchical Rank Pooling for Activity Recognition (2016, methodology): “Rank pooling is also used for temporal encoding at representation level [4, 5] or at image level leading to dynamic images [1].”
• Learning End-to-end Video Classification with Rank-Pooling (2016, methodology): “It captures the time varying information using rank pooling (Fernando et al., 2015; 2016b) and has the ability to handle variable length sequences.”
• Generalized Rank Pooling for Activity Recognition (2017, methodology): “Our experimental results show that the proposed scheme is significantly better at capturing the temporal structure of CNN features in action sequences compared to conventional pooling schemes or the basic form of rankpooling [14], while also achieving state-of-the-art performances.”
• UvA-DARE (Digital Academic Repository) Dynamic Image Networks for Action Recognition (2016, methodology): “Such observations were also made in [7].”
• Encouraging LSTMs to Anticipate Actions Very Early (2017, background): “While great progress has been made to recognize actions from complete sequences [16, 38, 7, 9, 30, 41, 31, 10, 2] in the past decade, action anticipation [25, 26, 24, 45, 32, 33, 21], which aims to predict the observed action as early as possible, has become a popular research problem only recently.”
• Person Re-identification: Past, Present and Future (2016, background): “Fernando et al. [118] propose a learning-to-rank model to capture how frame features evolve over time in a video, which yields video descriptors of video-wide temporal dynamics.”
• AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos (2016, background): “This could be the case since MIL is known to overfit as a result on focussing only on a single frame in a video [30, 19], while max pooling seems to fail to summarize relevant parts of an actions (and thus overfit) [7].”
• FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “The astounding performance of deep neural networks for action recognition [5, 43, 58, 12, 42, 56] are obtained by densely applying 2D [56, 31, 58, 13] or 3D [40, 5, 18, 12] models over video frames.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[56] proposed to represent depth sequences with three pairs of structured dynamic images [55] at the body, body part, and joint levels, which were then fed to CNNs followed by a score fusionmodule for fine-grainedHAR.”
• Deep Bilinear Learning for RGB-D Action Recognition (2018, methodology): “For example, [7] used a ranking machine to encode the dynamics among the sequential features.”
• Online Action Detection (2016, unknown): “, [1, 4, 5, 9, 13, 17]), the goal is to detect an action as it happens and ideally even before the action is fully completed.”
• Timeception for Complex Action Recognition (2018, methodology): “A widely used approach is statistical pooling: max and average pooling [18, 19], attention pooling [6], rank pooling [20], dynamic images [21] and context gating [7], to name a few.”