Learning Clip Representations for Skeleton-Based 3D Action Recognition

By Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, Farid Boussaïd. 2018

258 citations recorded by the source corpus.

This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a multitask convolutional neural network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and the feature learning method for 3D action recognition compared to the existing techniques.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. G. Wang
3. Hossein Rahmani
4. Ling-yu Duan
5. Alex ChiChung Kot
6. Qiuhong Ke
7. Amir Shahroudy
8. Hanqing Lu

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Hossein Rahmani
4. Qiuhong Ke
5. Alex ChiChung Kot
6. Amir Shahroudy
7. Ling-yu Duan
8. Hanqing Lu

In Representation & Self-Supervised Learning

1. Amir Shahroudy
2. G. Wang
3. Jun Liu
4. Jian Cheng
5. Qiuhong Ke
6. Hanqing Lu
7. Alex ChiChung Kot
8. Hossein Rahmani

In 3D Vision & Pose

1. Jun Liu
2. Amir Shahroudy
3. G. Wang
4. Alex ChiChung Kot
5. Ling-yu Duan
6. Qiuhong Ke
7. Zehua Sun
8. Jian Cheng

In Human-Computer Interaction

1. Jun Liu
2. Hossein Rahmani
3. Amir Shahroudy
4. G. Wang
5. Yifan Zhang
6. Jian Cheng
7. Hanqing Lu
8. Alex ChiChung Kot

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The methods in [30], [47], [48], [94], [102], [105] all use RNN/LSTM for 3D action recognition, and the approaches in [49], [103], [104], [106], [107] all use convolutional networks for spatial and temporal modeling.”
• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, unknown): “ST-LSTM[18] 55.7 57.9 GCA-LSTM[19] 61.2 63.3 RotClips+MTCNN[10] 62.2 61.8 SGN[35] 79.2 81.5 2s-AGCN[24] 82.9 84.9 Shift-GCN[4] 85.9 87.6 DC-GCN+ADG[3] 86.5 88.1 MS-G3D[20] 86.9 88.4 PA-ResGCN-B19 [26] 87.3 88.3 Dynamic GCN [34] 87.3 88.6 CTR-GCN (Bone Only) 85.7 87.5 CTR-GCN (Joint+Bone) 88.7 90.1…”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, background): “Table 4 compares non-graph [26, 27, 16, 28] and graph-based methods [33].”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “The idea of ‘clips’ is used in [218] as well.”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, background): “Methods X-sub X-setup FLOPs (G) Part-Aware LSTM [19] 25.5 26.3 - ST-LSTM [16] 55.7 57.9 - Multi CNN + RotClips [6] 62.2 61.8 - SkeMotion [17] 67.7 66.9 - TSRJI [1] 67.9 Our adaptive temporal shift graph convolution can adjust the receptive field adaptively and enjoy high efficiency.”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, unknown): “ccuracy (%). Method C-Subject C-Setup Year Dynamic Skeleton [7] 50.8 54.7 2015 PA-LSTM [23] 25.5 26.3 2016 ST-LSTM+TS [18] 55.7 57.9 2016 FSNet [17] 59.9 62.4 2019 2s-ALSTM [19] 61.2 63.3 2017 MT CNN [9] 62.2 61.8 2018 BPEM [20] 64.6 66.9 2018 2s-AGCN [26] 82.9 84.9 2019 MS-G3D Net [21] 86.9 88.4 20”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, unknown): “…89.4 4s DGNN (Shi et al. 2019a) 96.1 89.9 4s MS-AAGCN (Shi et al. 2019b) 96.2 90.0 2s MS-G3D (Liu et al. 2020) 96.2 91.5 4s Shift-GCN (Cheng et (Ke et al. 2018) 62.2 61.8 SkeMotion (Liu et al. 2017) 67.7 66.9 TSRJI (Caetano et al. 2019) 67.9 62.8 4s Shift-GCN (Cheng et al. 2020) 85.9 87.6 2s…”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, unknown): “3 RotClips+MTCNN [19] 2018 62.”