A New Representation of Skeleton Sequences for 3D Action Recognition

By Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, Farid Boussaïd. 2017

866 citations recorded by the source corpus.

This paper presents a new method for 3D action recognition with skeleton sequences (i.e., 3D trajectories of human skeleton joints). The proposed method first transforms each skeleton sequence into three clips each consisting of several frames for spatial temporal feature learning using deep neural networks. Each clip is generated from one channel of the cylindrical coordinates of the skeleton sequence. Each frame of the generated clips represents the temporal information of the entire skeleton sequence, and incorporates one particular spatial relationship between the joints. The entire clips include multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We propose to use deep convolutional neural networks to learn long-term temporal information of the skeleton sequence from the frames of the generated clips, and then use a Multi-Task Learning Network (MTLN) to jointly process all frames of the clips in parallel to incorporate spatial structural information for action recognition. Experimental results clearly show the effectiveness of the proposed new representation and feature learning method for 3D action recognition.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. G. Wang
3. Amir Shahroudy
4. Alex ChiChung Kot
5. Ling-yu Duan
6. Qiuhong Ke
7. Hossein Rahmani
8. Jian Cheng

In Biomedical Engineering

1. Jun Liu
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Liang Wang
6. G. Wang
7. Qiuhong Ke
8. Hossein Rahmani

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Dacheng Tao
4. Luc Van Gool
5. Liang Wang
6. Yi Yang
7. Xiaogang Wang
8. Ming-Hsuan Yang

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The methods [48], [49] are evaluated by using the suggested two evaluation criteria.”
• Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, neutral): “Most of the existing skeleton-based action recognition methods [30], [40], [45], [46], [47], [48] receive the fully observed segmented videos as input (each sample contains”
• Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “7 MTLN [24] - 79.”
• Learning Clip Representations for Skeleton-Based 3D Action Recognition (2018, methodology): “In [44], the clip representation is learned without taking viewpoint variations into account.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “In these methods [18, 58, 30, 51], long-term dependencies and semantic information are expected to be captured by the large receptive fields of deep networks.”
• Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Our ST-GCN is the first to apply graph CNNs to the task of skeleton based action recognition.”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “Deep learning methods usually rearrange a skeleton sequence as a pseudo-image [21,14,9] or a series of joint coordinates [32,50,25], then use CNNs or RNNs to predict action labels.”
• Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “[172] transformed each skeleton sequence”
• Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks (2019, methodology): “Early deep-learning-based approaches for skeleton-based action recognition manually structure the skeleton data as a sequence of joint-coordinate vectors [8], [9], [10], [11], [12], [13], [14] or as a pseudo-image [15], [16], [17], [18], [19], which is fed into RNNs or CNNs to”
• View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition (2018, background): “Considering the strong capability of convolutional neural networks for classification, several recent works [4], [20], [23], [30], [49] attempt to convert a skeleton sequence to 2D image(s) and then leverage convolutional neural networks for classification.”
• Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition (2018, methodology): “The RNN-based model has the capability to model the temporal dependency, but it is difficult to train the stacked RNN in practice [15, 17].”
• Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “With the development of deep-learning-based methods, the data-driven methods have become the mainstream [7, 25, 20, 28, 37, 18, 19, 14, 13, 21, 17, 16, 34, 29, 3].”
• Global Regularizer and Temporal-Aware Cross-Entropy for Skeleton-Based Early Action Recognition (2018, methodology): “image and extract one feature vector for computation efficiency, rather than extracting multi-features which are computational expensive as [10].”
• Skeleton-based action recognition with convolutional neural networks (2017, methodology): “Compared with [8], where an ImageNet pre-trained VGG19 net is used, our model is superior on its compact model size and fast inference speed as well.”
• SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Most of the existing 3D action recognition methods [63, 51, 57, 69, 24, 66, 10, 29] take fully observed segmented videos as input (each sample contains one full action instance), and output a class label.”
• Cross-Modal Knowledge Distillation for Action Recognition (2019, background): “More recently, most approaches use either recurrent neural networks to learn spatio-temporal features from sequences of skeleton data [14, 15, 12] or convolutional neural networks for classifying the skeleton sequences [16, 8, 17].”