NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

By Amir Shahroudy, Jun Liu, Tian-Tsong Ng, G. Wang. Computer Vision and Pattern Recognition, 2016

3,015 citations recorded by the source corpus.

Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+Dbased action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art handcrafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

View Paper: https://arxiv.org/abs/1604.02808

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “Twelve state-of-the-art 3D action recognition methods are evaluated on our dataset, namely, the Part-Aware LSTM [47], the Soft RNN [102], the Dynamic Skeleton [26], the Spatio-Temporal LSTM [48], the Internal Feature Fusion [94], the GCA-LSTM [30], the Multi-Task Learning Network [49], the FSNet…”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “In addition, we set images aside and conduct experiments on Action Recognition: to this aim, we introduce Split NTU60, a sequential classification benchmark built on top of the 3D skeletal data from the NTU-RGB+D dataset [50].”
Transformers in Vision: A Survey (2021, methodology): “The two streamed model achieves state-of-the-art results on NTU-RGB+D 60 and 120 [133, 175] datasets.”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, background): “The work of [53] introduced a new part-aware LSTM structure to discover the long-term motion patterns of skeleton-based body parts separately and learn the action classes based on these representations.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “Unlike the NTU RGB+D dataset [40], in which every video contains only one action, each video in the ChaLearn”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “For this experiment, we follow the two official evaluation protocols described in [37], i.”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, neutral): “6M [21] for respectively 2D and 3D pose estimation, and on Penn Action [59] and NTU RGB+D [44] for 2D and 3D action recognition, respectively.”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, positive): “Following [44], we sample 5% of the initial training set as a validation set, which is used for hyper-parameter optimization and for early stopping.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[109] proposed a part-aware LSTM human action learning model (P-LSTM) wherein memory is split across part-based cells.”
When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, methodology): “Comparative analysis We compared the proposed LogCOV-Net on the NTU-RGB+D dataset.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “Experiments on the NTU RG B + D Dataset The NTU RGB + D dataset [55] was collected with Kinect (V2).”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, methodology): “As shown in Table 8, our method surpasses the state-ofthe-art methods [39], [91], [92], [93], [94], [95], which demonstrates the effectiveness of our method in dealing with skeleton-based action recognition problem.”
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, methodology): “[37] propose a partaware extension of LSTM to utilize the physical structure”
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, methodology): “C V ] 1 4 A pr 2 activity recognition dataset, NTU-RGBD and obtain stateof-the-art results.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, background): “NTU RGB+D is a recent action recognition dataset that is quite large and provides depth and pose ground truth [35].”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, methodology): “NTU RGB+D Dataset (NTU) [30] — The largest dataset for human activity recognition has been acquired with a Kinect v2 sensor and contains more than 56K videos and 4 millions frames with 60 different activities including individual activities, interactions between 2 people and health related events.”
View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data (2017, methodology): “A general treatment employs a pre-processing step to transform the 3D joint coordinates from the camera coordinate system to a personcentric coordinate system by placing the body center at the origin, followed by rotating the skeleton such that the body plane is parallel to the (x, y)-plane, to make”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, methodology): “We also assign a consistent ID number over the entire dataset in a similar way in [29].”
A New Representation of Skeleton Sequences for 3D Action Recognition (2017, methodology): “An overall architecture of the proposed method is shown in Figure 1.”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, methodology): “An RNN is typically considered as memory cells, which are sensitive to both short as well as long term patterns.”