Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks

By Pichao Wang, Zhaoyang Li, Yonghong Hou, Wanqing Li. 2016

386 citations recorded by the source corpus.

Recently, Convolutional Neural Networks (ConvNets) have shown promising performances in many computer vision tasks, especially image-based recognition. How to effectively use ConvNets for video-based recognition is still an open problem. In this paper, we propose a compact, effective yet simple method to encode spatio-temporal information carried in 3D skeleton sequences into multiple 2D images, referred to as Joint Trajectory Maps (JTM), and ConvNets are adopted to exploit the discriminative features for real-time human action recognition. The proposed method has been evaluated on three public benchmarks, i.e., MSRC-12 Kinect gesture dataset (MSRC-12), G3D dataset and UTD multimodal human action dataset (UTD-MHAD) and achieved the state-of-the-art results.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “Most of the existing skeleton-based action recognition methods [30], [40], [45], [46], [47], [48] receive the fully observed segmented videos as input (each sample contains”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “The usage of RGB-D cameras is very popular for this case [44, 34, 56], as it allows for the use of articulated pose (skeletons) to be delivered in real time and”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “It has been shown that fine-tuning motion-based networks with spatial data (ImageNet) is more effective than training from scratch [56, 21, 51, 22].”
When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, methodology): “Last, but not least, in all works [5, 32, 38, 21, 19, 14, 15] the network adopted are deep, which actually makes the overall pipeline difficult to train, since overfitting must be controlled, the problem of local minima and saddle points must be faced, massive training data is required, as well as e”
A New Representation of Skeleton Sequences for 3D Action Recognition (2017, background): “ion changes and invariant to camera views [14]. With the prevalence of highlyaccurate and affordable devices, action recognition based on 3D skeleton sequence has been attracting increasing attention [49, 42, 6, 37, 54, 26, 22, 46, 19]. In this paper, we focus on skeleton-based 3D action recognition”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, methodology): “These methods either extracted features from one modality: depth [49, 60, 29, 59, 24, 51, 52] or RGB [27, 33] or skeleton [43, 55, 56, 4, 34, 19], or fuse the features extracted separately from them at a later stage [28, 16, 10, 58].”
Large-scale Isolated Gesture Recognition using Convolutional Neural Networks (2016, methodology): “The last category is to represent a video in one or multiple compact images and adopt available trained ConvNet architectures for finetuning [14], [15], [16], [23].”
Joint Distance Maps Based Action Recognition With Convolutional Neural Networks (2017, methodology): “Inspired by the works [21], [23], [24], where the authors used color to encode the temporal information, Hue is adopted in this letter to encode the variations of joint distances.”
Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN (2017, methodology): “…comparison on the UTD-MHAD dataset [11] Method Accuracy ELC-KSVD [31] 76.19% kinect & Inertial [30] 79.10% Cov3DJ [32] 85.58% SOS [33] 86.97% JTM [21] 87.90% Proposed Method 96.27% MSRC-12 [34] is a relatively large dataset for gesture/action recognition from 3D skeleton data captured by a…”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Most of the existing 3D action recognition methods [63, 51, 57, 69, 24, 66, 10, 29] take fully observed segmented videos as input (each sample contains one full action instance), and output a class label.”
Enhanced skeleton visualization for view invariant human action recognition (2017, methodology): “6 also compares our method with [20] and [21] , which can be considered as two speciﬁc cases of our visualization method.”
View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition (2018, methodology): “Some works [30], [49] use the 2D projection maps from the trajectories of joints to different orthogonal planes as the”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, neutral): “CS CV Deep learning on Lie Group [15] 61.37% 66.95% JTM+CNN [49] 73.40% 75.20% Res-TCN [22] 74.30% 83.10% SkeletonNet(CNN) [20] 75.94% 81.16% JDM+CNN [31] 76.20% 82.30% Clips+CNN+MTLN [21] 79.57% 84.83% Enhanced Visualization+CNN [33] 80.03% 87.21% 1 Layer RNN [46] 56.02% 60.24% 2 Layer RNN [46]…”
Deep Learning for Spatio-Temporal Data Mining: A Survey (2019, neutral): “Application domains Related works Transportation [177], [187], [116], [31], [29], [6], [32], [127], [53], [112], [26], [105], [166], [82], [23], [66], [108], [126], [81], [139], [78], [128], [146], [148], [72], [98], [189], [22], [24], [99], [96], [16], [97], [85], [15], [164], [165], [95], [140], [”
Large-scale Continuous Gesture Recognition Using Convolutional Neural Networks (2016, methodology): “recent development of deep learning, a few methods have also been developed based on Convolutional Neural Networks (ConvNets) [10], [11], [12], [13], [14] and Recurrent Neural Networks (RNNs) [15], [16], [17], [18].”
Occluded Prohibited Items Detection: An X-ray Security Inspection Benchmark and De-occlusion Attention Module (2020, background): “As the technology of deep learning [14] especially the convolutional neural network develops [3, 35], the recognition of occluded prohibited items from X-ray pictures can be regarded as an object detection problem of computer vision, which is widely studied in the literature.”