PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

By Chunhui Liu, Yueyu Hu, Yanghao Li, Sijie Song, Jiaying Liu. 2017

295 citations recorded by the source corpus.

Despite the fact that many 3D human activity benchmarks being proposed, most existing action datasets focus on the action recognition tasks for the segmented videos. There is a lack of standard large-scale benchmarks, especially for current popular data-hungry deep learning based methods. In this paper, we introduce a new large scale benchmark (PKU-MMD) for continuous multi-modality 3D human action understanding and cover a wide range of complex human activities with well annotated information. PKU-MMD contains 1076 long video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. Our dataset also provides multi-modality data sources, including RGB, depth, Infrared Radiation and Skeleton. With different modalities, we conduct extensive experiments on our dataset in terms of two scenarios and evaluate different methods by various metrics, including a new proposed evaluation protocol 2D-AP. We believe this large-scale dataset will benefit future researches on action detection for the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The G3D [45] and PKUMMD [46] datasets were introduced for activity analysis in continuous sequences, which respectively contain 210 and 1076 videos.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “The PKUMMD dataset [84] was captured for RGBD-based activity analysis in continuous sequences.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Sliding window-based methods [19] are simple extension of segmented-based action recognition methods.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “We followed [12] to calculate Mean Average Precision (mAP) and Mean Average Recall (mAR).”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “The MSRDailyActivity3D [237], UTD-MHAD [300], and PKU-MMD [28] datasets are also popularly used.”
Human Action Recognition and Prediction: A Survey (2018, background): “PKU-MMD Dataset (Chunhui et al., 2017) is a largescale multi-modal datasets focusing on long continuous sequences action detection and multi-modality action analysis.”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, methodology): “Experiments We evaluate our method on two challenging datasets: the OAD dataset [38] and the PKU-MMD dataset [40].”
Skeleton-based action recognition with convolutional neural networks (2017, background): “With the development of low-cost human skeleton capture systems (e.g. Kinect), large-scale 3D skeleton datasets have been made available [1, 2], which attract many research efforts [3] on skeleton-based human action recognition and detection.”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, methodology): “We evaluate the proposed Else-Net on two large-scale 3D skeleton-based human action recognition datasets: NTU RGB+D dataset [29] and PKU-MMD dataset [16].”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, background): “Action recognition datasets provide annotations of the action semantics, but they either contain no human pose labels [15, 88] or feature limited motion of daily activities [59, 60, 86].”
Graph Distillation for Action Detection with Privileged Modalities (2017, methodology): “We evaluate our method on two large-scale multimodal video benchmarks: NTU RGB+D [46] (classification) and PKU-MMD [28] (detection).”
Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation (2018, methodology): “The PKU-MMD dataset [ Liu et al. , 2017 ] is currently the largest skeleton-based action detection dataset.”
Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN (2017, background): “As an intrinsic high level representation, 3D skeleton is valuable and comprehensive for summarizing a series of human dynamics in the video, and thus benefits general action analysis [2].”
Skeleton boxes: Solving skeleton based action detection with a single deep convolutional neural network (2017, methodology): “For quantitative evaluation, we report errors obtained with the following error metrics, which have also been used in [1]. where I ∩ I ∗ denotes the intersection of the predicted and ground truth intervals and I ∪ I ∗ denotes their union.”