Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks

By Pichao Wang, Wanqing Li, Chuankun Li, Yonghong Hou. 2016

234 citations recorded by the source corpus.

Abstract Convolutional Neural Networks (ConvNets) have recently shown promising performance in many computer vision tasks, especially image-based recognition. How to effectively apply ConvNets to sequence-based data is still an open problem. This paper proposes an effective yet simple method to represent spatio-temporal information carried in 3 D skeleton sequences into three 2 D images by encoding the joint trajectories and their dynamics into color distribution in the images, referred to as Joint Trajectory Maps (JTM), and adopts ConvNets to learn the discriminative features for human action recognition. Such an image-based representation enables us to fine-tune existing ConvNets models for the classification of skeleton sequences without training the networks afresh. The three JTMs are generated in three orthogonal planes and provide complimentary information to each other. The final recognition is further improved through multiplicative score fusion of the three JTMs. The proposed method was evaluated on four public benchmark datasets, the large NTU RGB+D Dataset, MSRC-12 Kinect Gesture Dataset (MSRC-12), G3D Dataset and UTD Multimodal Human Action Dataset (UTD-MHAD) and achieved the state-of-the-art results.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Christian Wolf
2. Julien Mille
3. Fabien Baradel

In Computer Vision and Pattern Recognition

1. Julien Mille
2. Fabien Baradel
3. Christian Wolf
4. Jun Liu
5. Andrew Zisserman
6. Alan L. Yuille
7. Cordelia Schmid
8. Luc Van Gool

In Human-Computer Interaction

1. Christian Wolf
2. Julien Mille
3. Fabien Baradel
4. David Picard
5. Hedi Tabia
6. Diogo Carbonera Luvizon
7. Jun Liu

Citations in the corpus

• Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “The usage of RGB-D cameras is very popular for this case [44, 34, 56], as it allows for the use of articulated pose (skeletons) to be delivered in real time and relatively cheaply by some middleware.”
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, methodology): “Performance close to stateof-the-art is achieved on the smaller MSR Daily Activity 3D dataset.”