Deep Learning on Lie Groups for Skeleton-Based Action Recognition

By Zhiwu Huang, Chengde Wan, Thomas Probst, Luc Van Gool. 2016

296 citations recorded by the source corpus.

In recent years, skeleton-based action recognition has become a popular 3D classification problem. State-of-the-art methods typically first represent each motion sequence as a high-dimensional trajectory on a Lie group with an additional dynamic time warping, and then shallowly learn favorable Lie group features. In this paper we incorporate the Lie group structure into a deep network architecture to learn more appropriate Lie group features for 3D action recognition. Within the network structure, we design rotation mapping layers to transform the input Lie group features into desirable ones, which are aligned better in the temporal domain. To reduce the high feature dimensionality, the architecture is equipped with rotation pooling layers for the elements on the Lie group. Furthermore, we propose a logarithm mapping layer to map the resulting manifold data into a tangent space that facilitates the application of regular output layers for the final classification. Evaluations of the proposed network for standard 3D human action recognition datasets clearly demonstrate its superiority over existing shallow Lie group feature learning methods as well as most conventional deep learning methods.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, methodology): “Huang et al. (2017) deﬁne a network where the intermediate activations of the network are 3D rotations representing skeletal poses and embed elements into the Lie algebra using the log map.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, neutral): “Most of the existing skeleton-based action recognition methods [30], [40], [45], [46], [47], [48] receive the fully observed segmented videos as input (each sample contains”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Gaming 3D dataset (G3D) [29](http://dipersec.king.ac.uk/G3D/) captured by Kingston University in 2012 focuses on real-time action recognition in gaming scenario.”
When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, methodology): “If compared to [32, 21, 15, 5], the training time in this case is much longer even if using GPU acceleration.”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, neutral): “Skeleton-based action recognition has been explored in different aspects in recent years [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57].”
Explainability Methods for Graph Convolutional Neural Networks (2019, methodology): “In [36] GCNNs were used for shape segmentation, and in [14], they were used for skeleton-based action recognition.”
Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition (2018, background): “When increasing K 1 , K 2 , the local ﬁltering region will cover (Vemulapalli, Arrate, and Chellappa 2014) 52.76% 50.08% Dynamic Skeletons (Hu et al. 2015) 65.22% 60.23% HBRNN (Du, Wang, and Wang 2015) 63.97% 59.07% LieNet (Huang et al. 2016) 66.95% 61.37% Deep LSTM (Shahroudy et al. 2016) 67.29% 60”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Most of the existing 3D action recognition methods [63, 51, 57, 69, 24, 66, 10, 29] take fully observed segmented videos as input (each sample contains one full action instance), and output a class label.”
Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition (2018, unknown): “LieNet-3Blocks [21] 61.”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, methodology): “Method CS CV Deep learning on Lie Group [15] 61.”
Federated Transfer Learning for EEG Signal Classification (2020, background): “The paradigm of deep neural networks on SPD matrices has been proposed to interpolate, manipulate and classify SPD matrices in bunch of application areas [14], [15], [16], [17].”
Coding Kendall's Shape Trajectories for 3D Action Recognition (2018, background): “In the literature, two alternatives have been proposed to overcome these problems for different Riemannian manifolds – they are either Extrinsic [17, 20, 23, 27] or Intrinsic [4, 5, 21, 22].”
Action2Motion: Conditioned Generation of 3D Human Motions (2020, methodology): “t. For motion representation, the theory of Lie algebra has proven its effectiveness for articulate pose modeling in related tasks including pose estimation, action recognition, and motion prediction [8, 11, 20, 31]. Specifically, human pose is characterized as a kinematic tree based on physics prin”
Learning Action Recognition Model from Depth and Skeleton Videos (2017, methodology): “We compare our performance to the state-of-the-art action recognition methods including Comparative Coding Descriptor (CCD) [5], Histogram of Oriented Gradients (HOG(2)) [14], Discriminative Virtual Views (DVV) [11], Continuous Virtual Path (CVP) [41], Histogram of Oriented 4D Normals (HON4D) [15], ”
Recognizing Human Actions as the Evolution of Pose Estimation Maps (2018, methodology): “, CNN [17, 20], RNN [9] and LSTM [26, 25], have achieved high performances on the extracted 3D poses, which outperform methods [32, 50] that rely on raw depth video sequences.”