Multimodal Multipart Learning for Action Recognition in Depth Videos

By Amir Shahroudy, Gang Wang, Tian-Tsong Ng, Qingxiong Yang. 2015

125 citations recorded by the source corpus.

The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart features are formulated into the learning framework via the proposed hierarchical mixed norm, to regularize the structured features of each part and to apply sparsity between them, in favor of a group feature selection. Our experimental results expose the effectiveness of the proposed learning method in which it outperforms other methods in all three tested datasets while saturating one of them by achieving perfect accuracy.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “Each pair contains very closely re-lated actions with differences along temporal axis e.g . pick up/put down a box, push/pull a chair, wear/take off a hat, etc. State-of-the-art methods [17, 32, 31] achieved perfect accuracy on this benchmark.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “The work of [38] proposed hierarchical mixed norms to fuse different features and select most informative body parts in a joint learning framework.”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “Our method ties with two recent works (MMMP [54], and BHIM [22]) in saturating the benchmark by achieving the flawless 100% accuracy on this dataset.”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, background): “Fusing RGB-D information with skeleton data to build a multisensory representation has the potential to address this problem [107], [126], [191] and improve the descriptive power of the existing space-time human representations.”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, neutral): “Skeleton-based action recognition has been explored in different aspects in recent years [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57].”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “Furthermore, many methods [12, 30] divide the human body into several parts and learn the co-occurrence information, respectively.”
Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks (2016, background): “ by extracting spatial descriptor [30], [56]– poses [35], [61]–[63]. The groups of joints aim to detect the discriminative subsets of joints to differentiate actions. Methods such as [33], [61], [64]–[67] focus on mining the subsets of most discriminative joints or consider the correlation of predeﬁ”
Learning a Deep Model for Human Action Recognition from Novel Viewpoints (2016, methodology): “To overcome this problem, some techniques infer 3D scene structure and use geometric transformations to achieve view invariance [3], [24], [25], [26], [27], [28], [29], [30], [31], [32].”
Learning Action Recognition Model from Depth and Skeleton Videos (2017, background): “[27] proposed hierarchical mixed norms to fuse different features and select the most informative body joints.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Therefore, early methods [351], [352] have attempted to fuse hand-crafted features extracted from both the skeleton and the depth sequences for more accurate HAR.”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “Having multiple data streams has led to the study of combining different sources of information such as depth and pose [4, 40, 52, 69], color and pose [88], and all of them [19].”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, methodology): “After the development of cheap and easy-to-use depth sensors, such as Kinect and XTion, human action recognition in 3D skeleton sequences becomes very popular [1, 79], and a series of hand-crafted features [72, 63, 75, 64, 52, 13, 70, 67] and deep learning based approaches [11, 83, 51, 30, 42, 26] h”
Sensor-based and vision-based human activity recognition: A comprehensive survey (2020, methodology): “Furthermore, DFT [46,176] , which extracts FD data from the ROI, is also considered a global features representation method.”