PoTion: Pose MoTion Representation for Action Recognition

By Vasileios Choutas, Philippe Weinzaepfel, Jérôme Revaud, Cordelia Schmid. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

305 citations recorded by the source corpus.

Most state-of-the-art methods for action recognition rely on a two-stream architecture that processes appearance and motion independently. In this paper, we claim that considering them jointly offers rich information for action recognition. We introduce a novel representation that gracefully encodes the movement of some semantic keypoints. We use the human joints as these keypoints and term our Pose moTion representation PoTion. Specifically, we first run a state-of-the-art human pose estimator [4] and extract heatmaps for the human joints in each frame. We obtain our PoTion representation by temporally aggregating these probability maps. This is achieved by 'colorizing' each of them depending on the relative time of the frames in the video clip and summing them. This fixed-size representation for an entire video clip is suitable to classify actions using a shallow convolutional neural network. Our experimental evaluation shows that PoTion outperforms other state-of-the-art pose representations [6, 48]. Furthermore, it is complementary to standard appearance and motion streams. When combining PoTion with the recent two-stream I3D approach [5], we obtain state-of-the-art performance on the JHMDB, HMDB and UCF101 datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

View Paper: https://www.semanticscholar.org/paper/6009bba115904bc3bf876224db90b232c4f0a48f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

Omni-sourced Webly-supervised Learning for Video Recognition (2020, unknown): “ream [39] ImageNet 88.0 59.4 TSN [48] ImageNet 94.2 69.4 RGB-I3D[3] ImageNet + Kinetics 95.6 74.8 Flow-I3D[3] ImageNet + Kinetics 96.7 77.1 Two-Stream-I3D[3] ImageNet + Kinetics 98.0 80.7 I3D + PoTion[6] ImageNet + Kinetics 98.2 80.9 I3D + PA3D[51] ImageNet + Kinetics / 82.1 SlowOnly-8x8-R101 Kineti”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “e characterized by the individual actions a person takes in interaction with the surrounding objects and the environment. A signiﬁcant amount of research has focused on visual understanding of humans [8, 25,36,40,59,65,70,74] and objects [4,28,64,66], in isolation from each other. However, the probl”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, methodology): “Recently, an intermediate representation ( e.g . human skeleton in [48, 4, 5]) is used by several methods, which could be described as the third pipeline.”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “Simultaneously, there are approaches that use features extracted from body pose information, e.g ., Po-Tion [12] and JMRN [68].”
Evolving Space-Time Neural Architectures for Videos (2018, unknown): “9 PoTion [2] 80.”
Revisiting Skeleton-based Action Recognition (2021, background): “PoTion [5] aggregates heatmaps along the temporal dimension with color encodings to get the pose motion representation, while PA3D [38] does the aggregation with 1×1 convolutions.”
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, background): “With the introduction of robust body key-points (socalled skeleton) detection [69], some studies try to solve human action recognition by body joints only [18, 73] or use body joints along with the image stream [13].”
Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition With CNNs (2019, methodology): “Such a fusion of CNNs and handcrafted representations is timeconsuming due to pre-processing, descriptor extraction, encoding and tuning parameters.”