Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation

By C. Li, Qiaoyong Zhong, Di Xie, Shiliang Pu. International Joint Conference on Artificial Intelligence, 2018

579 citations recorded by the source corpus.

Skeleton-based human action recognition has recently drawn increasing attentions with the availability of large-scale skeleton datasets. The most crucial factors for this task lie in two aspects: the intra-frame representation for joint co-occurrences and the inter-frame representation for skeletons' temporal evolutions. In this paper we propose an end-to-end convolutional co-occurrence feature learning framework. The co-occurrence features are learned with a hierarchical methodology, in which different levels of contextual information are aggregated gradually. Firstly point-level information of each joint is encoded independently. Then they are assembled into semantic representation in both spatial and temporal domains. Specifically, we introduce a global spatial aggregation scheme, which is able to learn superior joint co-occurrence features over local aggregation. Besides, raw skeleton coordinates as well as their temporal difference are integrated with a two-stream paradigm. Experiments show that our approach consistently outperforms other state-of-the-arts on action recognition and detection benchmarks like NTU RGB+D, SBU Kinect Interaction and PKU-MMD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1b10e59adfa3f0f7748f48e2e64e54db2a5362d3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “nd 89.2% in terms of two protocols on the NTU dataset. To demonstrate the effectiveness of our method, we choose the following related methods to compare and analyze the results: AGC-LSTM vs HCN. HCN [13] employs the CNN model for learning global co-occurrences from skeleton data. It treats each joi”
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, background): “In an earlier work [13], a Poster Session B1: Deep Learning for Multimedia MM '20, October 12–16, 2020, Seattle, WA, USA”
Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “In an earlier work [29], a convolutional co-occurrence feature learning framework is proposed, where a hierarchical methodology is employed to gradually aggregate different levels of contextual information.”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “The ﬁrst part of the table shows our unimodal baselines with I3D on RGB and HCN [53] on skeletons.”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, methodology): “ i.e. 91.5% and 96.0% in the C-Subject and C-View settings of NTU-RGB+D respectively. Although our accuracy is on par with MS-G3D Net [21], our model is much more efficient as analyzedinTable5.[33]and[14]aretworepresentativemethodsfor GCN-based and CNN-based methods. Dynamic GCN outperforms them by ”
Cross-Modal Knowledge Distillation for Action Recognition (2019, methodology): “All other hyperparameters are the same as in [6] and [7].”
Human Action Recognition From Various Data Modalities: A Review (2020, unknown): “Thoker and Gall [360] utilized TSN [50] as the teacher network taking RGB videos, and an ensemble of student networks (e.g., ST-GCN [152] and HCN [174]) handling skeleton data, to perform knowledge distillation for cross-modality HAR.”