Fine-Grained Activity Recognition with Holistic and Pose Based Features

By Leonid Pishchulin, Mykhaylo Andriluka, Bernt Schiele. 2014

54 citations recorded by the source corpus.

Holistic methods based on dense trajectories [29, 30] are currently the de facto standard for recognition of human activities in video. Whether holistic representations will sustain or will be superseded by higher level video encoding in terms of body pose and motion is the subject of an ongoing debate [12]. In this paper we aim to clarify the underlying factors responsible for good performance of holistic and pose-based representations. To that end we build on our recent dataset [2] leveraging the existing taxonomy of human activities. This dataset includes \(24,920\) video snippets covering \(410\) human activities in total. Our analysis reveals that holistic and pose-based methods are highly complementary, and their performance varies significantly depending on the activity. We find that holistic methods are mostly affected by the number and speed of trajectories, whereas pose-based methods are mostly influenced by viewpoint of the person. We observe striking performance differences across activities: for certain activities results with pose-based features are more than twice as accurate compared to holistic features, and vice versa. The best performing approach in our comparison is based on the combination of holistic and pose-based approaches, which again underlines their complementarity.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “Recent work on activity recognition in still images relies on a variety of cues, such as pose and global context [15, 23, 26].”
Contextual Action Recognition with R*CNN (2015, methodology): “A direct comparison with published results, as shown in Figure 1(b) in [25], can be drawn.”
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (2016, background): “Recent work on activity recognition in still images relies on a variety of cues, such as pose and global context [27, 24].”
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (2016, methodology): “We train deep networks on the HICO [14] and MPII [15] datasets to predict human activity.”
Multi-cue Zero-Shot Learning with Strong Supervision (2016, background): “The notion of semantic parts plays a central role in domains such as human pose estimation [49, 6], action recognition [9, 37, 7] and face detection [55].”
ActivityNet: A large-scale video benchmark for human activity understanding (2015, methodology): “The MPII Human Pose Dataset [2] focuses on human pose estimation, and was recently applied to action recognition [29].”
Attentional Pooling for Action Recognition (2017, methodology): “…problems, including action recognition, have seen a large boost with the recent advancements in deep learning (e.g., MPII performance went up from 5% mAP [34] to 27% mAP [18]), video based recognition still relies on hand crafted features such as iDT [53] to obtain competitive performance.”
Pose for Action - Action for Pose (2016, background): “More recently, [10] proposes a convolutional recurrent neural network architecture that takes as input a set of video frames and sequentially estimates the body part locations in each frame, while also using the information of estimated body parts in previous frames.”
2D Human Pose Estimation: New Benchmark and State of the Art Analysis (2014, methodology): “Activity recognition We evaluate the performance of the state-of-the-art activity recognition methods on our benchmark in [4].”
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation (2016, methodology): “Holistic Features: Holistic methods using spatio-temporal features with a bag of words representation are standard for large-scale [1,20,21,22] and ﬁne-grained [4,5,6,21,23] action analysis.”