Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

By Amir Shahroudy, Tian-Tsong Ng, Yihong Gong, G. Wang. 2016

248 citations recorded by the source corpus.

Single modality action recognition on RGB or depth sequences has been extensively explored recently. It is generally accepted that each of these two modalities has different strengths and limitations for the task of action recognition. Therefore, analysis of the RGB+D videos can help us to better study the complementary properties of these two types of modalities and achieve higher levels of performance. In this paper, we propose a new deep autoencoder based shared-specific feature factorization network to separate input multimodal signals into a hierarchy of components. Further, based on the structure of the features, a structured sparsity learning machine is proposed which utilizes mixed norms to apply regularization within components and group selection between them for better classification performance. Our experimental results show the effectiveness of our cross-modality feature analysis framework by achieving state-of-the-art accuracy for action classification on five challenging benchmark datasets.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Multimodal Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Human-Computer Interaction

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “Each pair contains very closely re-lated actions with differences along temporal axis e.g . pick up/put down a box, push/pull a chair, wear/take off a hat, etc. State-of-the-art methods [17, 32, 31] achieved perfect accuracy on this benchmark.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “In addition to the skeleton data, multimodal approaches can also benefit from the visual cues [45].”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “DSSCA - SSLM [45] 74.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, neutral): “(Shahroudy et al., 2017) RGB+Depth DSSCA-SSLM Hierarchical fusion 97.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, background): “Though a series of deep learning based models have been proposed for video analysis in existing works [63], [64], most of them did not consider the attention mechanism.”
Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks (2019, methodology): “Methods Pose RGB CS (%) CV (%) DSSCA-SSLM [47] X X 74.”
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference (2018, background): “Multimodal methods which combine optical ﬂow or depth information along with RGB data [41, 39] are shown to provide state-of-the-art results for generic (not just human) activity recognition tasks.”
Graph Distillation for Action Detection with Privileged Modalities (2017, methodology): “ies and tested on the single modality specied in the table. The available modalities are RGB, depth (D), optical ow (F), and skeleton (S). Method Test Modality mAP Method Test Modality mAP Shahroudy [47] RGB+D 0.749 Ours RGB 0.895 Liu [29] RGB+D 0.775 Ours D 0.875 Liu [32] S 0.800 Ours F 0.857 Ding ”
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition (2019, methodology): “We have conducted experiments to compare with state-of-theart RGBD-based methods DSSCA [15] and MMUDL [66] 4.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[327] focused on studying the correlation between modalities, and factorizing them into their correlated and independent components.”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, methodology): “[205] constructed a unimodal representation using a hierarchical factorization scheme that is limited to representing redundant feature parts and other completely orthogonal parts.”
Progressive Teacher-Student Learning for Early Action Prediction (2019, background): “Some researchers found that combining the multimodality features extracted from RGB, depth and skeleton sequences can capture more useful action information and obtain a better recognition performance [45] [17] [35] [31].”
Learning a Deep Model for Human Action Recognition from Novel Viewpoints (2016, methodology): “To overcome this problem, some techniques infer 3D scene structure and use geometric transformations to achieve view invariance [3], [24], [25], [26], [27], [28], [29], [30], [31], [32].”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, methodology): “Although deep learning based methods [40, 36, 55] have been used for action recognition in existing works, most of them do not focus on attention.”
Deep Bilinear Learning for RGB-D Action Recognition (2018, methodology): “To mine more interactive information among multi-modal features, lots of methods are proposed to explicitly learn shared-specific structures among features [11, 13, 28].”