NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

By Jun Liu, Amir Shahroudy, Mauricio Perez, G. Wang, Ling-yu Duan, Alex ChiChung Kot. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

1,730 citations recorded by the source corpus.

Research on depth-based human activity analysis achieved outstanding performance and demonstrated the effectiveness of 3D representation for action recognition. The existing depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of large-scale training samples, realistic number of distinct class categories, diversity in camera views, varied environmental conditions, and variety of human subjects. In this work, we introduce a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. We evaluate the performance of a series of existing 3D activity analysis methods on this dataset, and show the advantage of applying deep learning methods for 3D-based human action recognition. Furthermore, we investigate a novel one-shot 3D activity recognition problem on our dataset, and a simple yet effective Action-Part Semantic Relevance-aware (APSR) framework is proposed for this task, which yields promising results for recognition of the novel action classes. We believe the introduction of this large-scale dataset will enable the community to apply, adapt, and develop various data-hungry learning techniques for depth-based and RGB+D-based human activity understanding.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

View Paper: https://arxiv.org/abs/1905.04757

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…WebSight [122] Type: Video Datasets Captioning Mementos [254], ShareGPT4Video [30], VideoGPT+ [174], ShareGPT4o-Video [35] General QA VideoChat2-IT [131], EgoTaskQA [99], NTU RGB+D [152], CLEVRER [276], STAR [259], LSMDC [201] Table 4: Summary of the pre-training data mixture of InternVL 2.5.”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, background): “NTU-120 RGB+D: NTU-120 RGB+D (Liu et al. 2019) dataset is the currently largest 3D skeleton-based action recognition dataset captured under various environmental conditions.”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, result): “We also compare the scale of our dataset with NTU RGB+D 60 [29] and NTU RGB+D 120 [20], which are the largest full-body pose dataset.”
Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “To verify the superiority and generality of our approach, the shift GCN is compared with state-of-the-art methods on three datasets: NTU RGB+D dataset [19], Northwestern-UCLA dataset [29], and the recent proposed NTU-120 RGB+D dataset [15], shown in Table 5, Table 6, and Table 7 respectively.”
Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, background): “Note that there are 302 wrong samples selected by [28] that need to be ignored.”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Comparative results on NTU RGB+D 60 [42] ( left ), NTU RGB+D 120 [30] ( middle ), and NW-UCLA [55] ( right ).”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, background): “Various skeleton-based action recognition approaches [5, 29, 13, 17, 20, 18, 19, 30, 42, 15, 27, 31, 39, 14] have been proposed.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “RGB+D [161], NTU RGB+D 120 [151], and MMAct [299] are large benchmark datasets suitable for fusion and co-learning of different modalities.”
Skeleton-based action recognition via spatial and temporal transformer networks (2020, methodology): “…for a comparison with the state-of-the-art, we test the resulting best conﬁgurations on the Kinetics dataset (Kay et al. (2017)) and on the NTU-RGB + D 120 dataset (Liu et al. (2019)), which represents to date one of the most complex skeleton-based action recognition benchmarks (see Table 4-5).”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, result): “NTU120 RGB+D Dataset (NTU120) [25].”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, background): “NTU RGB+D 120 [17] is currently the largest dataset with 3D joints annotations for human action recognition, which extends NTU RGB+D with additional 57,367 skeleton sequences over 60 extra action classes.”
Human Action Recognition and Prediction: A Survey (2018, methodology): “Recently, it has been extended to 120 action classes and another 114, 480 video samples in [154].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “MUGL [Ma-heshwari et al. 2022] model followed with elaborated Conditional Gaussian-Mixture-VAE [Dilokthanakul et al. 2016] that supports up to 120 classes and multi-person generation, based on the NTU-RGBD-120 dataset [Liu et al. 2019].”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “We further expand the training set by including datasets from PerceptionTestQA [Patraucean et al., 2024], TVQA [Lei et al., 2018], NTU-RGB-D [Liu et al., 2020], and EgotaskQA [Grauman et al., 2022], along with grounding datasets based on DiDeMo [Anne Hendricks et al., 2017b] and COCO [Lin et al.,…”
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “[16] recently.”