Skeleton-based action recognition with convolutional neural networks

By C. Li, Qiaoyong Zhong, Di Xie, Shiliang Pu. 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2017

417 citations recorded by the source corpus.

Current state-of-the-art approaches to skeleton-based action recognition are mostly based on recurrent neural networks (RNN). In this paper, we propose a novel convolutional neural networks (CNN) based framework for both action classification and detection. Raw skeleton coordinates as well as skeleton motion are fed directly into CNN for label prediction. A novel skeleton transformer module is designed to rearrange and select important skeleton joints automatically. With a simple 7-layer network, we obtain 89.3% accuracy on validation set of the NTU RGB+D dataset. For action detection in untrimmed videos, we develop a window proposal network to extract temporal segment proposals, which are further classified within the same network. On the recent PKU-MMD dataset, we achieve 93.7% mAP, surpassing the baseline by a large margin.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/d1830fc99293b54658eb22df06f6eb3d48b7847e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Liang Wang
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Zhang Zhang
6. Caifeng Shan
7. Yisheng Song
8. Lei Shi

In Computer Vision and Pattern Recognition

1. Liang Wang
2. Luc Van Gool
3. Hanqing Lu
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Yi Yang
8. Li Fei-Fei

In Human-Computer Interaction

1. Yifan Zhang
2. Hanqing Lu
3. Jian Cheng
4. Liang Wang
5. Caifeng Shan
6. Yisheng Song
7. Zhang Zhang
8. Lei Shi

Citations in the corpus

• Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition (2018, methodology): “[20] employed a two-stream CNN architecture to combine the position and velocity information of human joints.”
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, positive): “[12] design a simple yet effective CNN architecture for action classification from trimmed skeleton sequences.”
• Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, methodology): “[13] treats a skeleton sequence as a pseudo-image, in which the coordinate of each skeleton joint (x ,y, z) is regarded as three channels of the image.”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “0 3scale ResNet 152 [10] 90.”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, neutral): “…Ind-RNN [20] 81.8 88.0 2-Stream 3DCNN [21] 66.8 72.6 TCN [11] 74.3 83.1 ClipCNN+MTLN [9] 79.6 84.8 Synthesized CNN [27] 80.0 87.2 CNN+Motion+Trans [15] 83.2 88.8 ST-GCN [46] 81.5 88.3 Motif+VTDB [44] 84.2 90.2 STGR-GCN [13] 86.9 92.3 AS-GCN [18] 86.8 94.2 Non-local adaptive GCN [34] 88.5 95.1…”
• Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “With the development of deep-learning-based methods, the data-driven methods have become the mainstream [7, 25, 20, 28, 37, 18, 19, 14, 13, 21, 17, 16, 34, 29, 3].”
• Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation (2018, methodology): “3 Methods In all previous CNN-based methods [ Du et al. , 2016; Ke et al. , 2017; Li et al. , 2017b ] , joint coordinates are speci-fied as channels.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, background): “The same is true for skeleton-based action recognition [6, 22, 18, 3].”
• Cross-Modal Knowledge Distillation for Action Recognition (2019, background): “Action recognition is addressed in many works and in particular deep learning methods have been proposed for various modalities like RGB videos [1, 2, 3, 4] or skeleton data [5, 6, 7, 8].”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “These methods manually construct the skeleton sequence as a kind of grid-shape structure such as a pseudo-image (Ke et al. 2017; Soo Kim and Reiter 2017; Li et al. 2017) or a sequence of the coordinate vectors (Du et al. 2015; Zhang et al. 2017; Liu et al. 2016).”
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, background): “Our ST-GCN is the first to apply graph CNNs to the task of skeleton based action recognition.”
• Graph Distillation for Action Detection with Privileged Modalities (2017, methodology): “We use a total of six modalities in our experiments: RGB, depth (D), optical flow (F), and three skeleton features (S) named Joint-Joint Distances (JJD), Joint-Joint Vector (JJV), and Joint-Line Distances (JLD) [9,24], respectively.”
• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, positive): “[8] design a simple yet effective CNN architecture for action classification from trimmed skeleton sequences.”