Interpretable 3D Human Action Analysis with Temporal Convolutional Networks

By Tae Soo Kim, Austin Reiter. 2017

661 citations recorded by the source corpus.

The discriminative power of modern deep learning models for 3D human action recognition is growing ever so potent. In conjunction with the recent resurgence of 3D human action representation with 3D skeletons, the quality and the pace of recent progress have been significant. However, the inner workings of state-of-the-art learning based methods in 3D human action recognition still remain mostly black-box. In this work, we propose to use a new class of models known as Temporal Convolutional Neural Networks (TCN) for 3D human action recognition. TCN provides us a way to explicitly learn readily interpretable spatio-temporal representations for 3D human action recognition. Through this work, we wish to take a step towards a spatio-temporal model that is easier to understand, explain and interpret. The resulting model, Res-TCN, achieves state-of-the-art results on the largest 3D human action recognition dataset, NTU-RGBD.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “HMDB-51 (accuracy over 3 splits) Method Accuracy Temporal segment networks[54] 69.4% AdaScan[19] 54.9% AdaScan + IDT + C3D[19] 66.9% ST ResNet[10] 66.4% ST ResNet + IDT[10] 70.3% ST Multiplier Network[11] 68.9% ST Multiplier Network + IDT[11] 72.2% Two-stream I3D[5] 66.4% Two-stream I3D+ (Kinetics 3”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Kim and Reiter [67] approached the problem differently and proposed to use the Temporal Convolutional Neural Networks (TCN) [71] for skeleton based action recognition.”
Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, methodology): “2015), CNN-based methods (Kim and Reiter 2017; Liu, Liu, and Chen 2017), LSTM-based methods (Shahroudy et al.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “Kim and Reiter [149]”
Modality Distillation with Multiple Stream Networks for Action Recognition (2018, neutral): “2.3% 4 Ours - depth model, step 1 Depth 70.44% 75.16% Ours - RGB model, step 1 RGB 66.52% 71.39% Deep RNN [9] Joints 56.3% 64.1% Deep LSTM [9] Joints 60.7% 67.3% Sharoudy [9] Joints 62.93% 70.27% Kim [19] Joints 74.3% 83.1% Sharoudy [12] RGB+D 74.86% - Liu [13] RGB+D 77.5% 84.5% Ours - step 2 RGB+D ”
Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “With the development of deep learning, researchers manually structure the skeleton data as a pseudo-image [5, 7, 10, 14] or a sequence of coordinate vectors [2, 16, 19, 35, 36], which is fed into CNNs or RNNs”
Classifying Signals on Irregular Domains via Convolutional Cluster Pooling (2019, neutral): “9 Temporal Conv [12] 74.”
Cross-Modal Knowledge Distillation for Action Recognition (2019, background): “More recently, most approaches use either recurrent neural networks to learn spatio-temporal features from sequences of skeleton data [14, 15, 12] or convolutional neural networks for classifying the skeleton sequences [16, 8, 17].”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “CNN-based models which try to embed the motion patterns into a skeleton image also achieve remarkable results (Ke et al. 2017; Soo Kim and Reiter 2017).”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, neutral): “CS CV Deep learning on Lie Group [15] 61.37% 66.95% JTM+CNN [49] 73.40% 75.20% Res-TCN [22] 74.30% 83.10% SkeletonNet(CNN) [20] 75.94% 81.16% JDM+CNN [31] 76.20% 82.30% Clips+CNN+MTLN [21] 79.57% 84.83% Enhanced Visualization+CNN [33] 80.03% 87.21% 1 Layer RNN [46] 56.02% 60.24% 2 Layer RNN [46]…”
Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition (2019, methodology): “Specifically, STGCN [29] combines GCN with temporal CNN to capture spatio-temporal features, and SR-TSL [23] use gated recurrent unit (GRU) to propagate messages on graphs and use LSTM to learn the temporal features.”
Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, neutral): “…[39] 50.1 52.8 STA-LSTM [36] 73.4 81.2 VA-LSTM [48] 79.2 87.7 ARRN-LSTM [16] 80.7 88.8 Ind-RNN [20] 81.8 88.0 2-Stream 3DCNN [21] 66.8 72.6 TCN [11] 74.3 83.1 ClipCNN+MTLN [9] 79.6 84.8 Synthesized CNN [27] 80.0 87.2 CNN+Motion+Trans [15] 83.2 88.8 ST-GCN [46] 81.5 88.3 Motif+VTDB [44] 84.2…”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, neutral): “ours) 91.5 96.0 Table 7: Performance comparison on the Skeleton-Kinetics dataset. Both top-1 and top-5 accuracies (%) are reported. Method Top-1 Acc. Top-5 Acc. Year Deep LSTM [23] 16.4 35.3 2016 TCN [10] 20.3 40.0 2017 ST-GCN [33] 30.7 52.8 2018 STGR-GCN [12] 33.6 56.1 2019 AS-GCN [15] 34.8 56.5 20”
Temporal convolutional neural (TCN) network for an effective weather forecasting using time-series data from the local weather station (2020, methodology): “Recently, Temporal Convolutional Neural Network (TCN), Recurrent Neural Networks (RNN) and deep learning have attracted considerable attention due to their superior performance (Jozefowicz et al., 2015; Kim and Reiter, 2017).”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, methodology): “After the development of cheap and easy-to-use depth sensors, such as Kinect and XTion, human action recognition in 3D skeleton sequences becomes very popular [1, 79], and a series of hand-crafted features [72, 63, 75, 64, 52, 13, 70, 67] and deep learning based approaches [11, 83, 51, 30, 42, 26] h”