RGB-D-based action recognition datasets: A survey

By Jing Zhang, Wanqing Li, Philip O. Ogunbona, Pichao Wang, Chang-Fu Tang. 2016

264 citations recorded by the source corpus.

Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-a-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols. HighlightsA detailed review and in-depth analysis of 44 publicly available RGB-D-based action datasets.Recommendations on the selection of datasets and evaluation protocols for use in future research.Identification of some limitations of these datasets and evaluation protocols.Recommendations on future creation of datasets and use of evaluation protocols.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Human-Computer Interaction

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “It is worth mentioning, there are more than 40 datasets speciﬁcally for 3D human action recognition [47].”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Readers are referred to these survey papers [31], [32], [33], [34], [35], [36] for a more extensive list of the current 3D activity analysis datasets and methods.”
Recent advances in convolutional neural networks (2015, background): “Applying CNNs on videos is challenging because traditional CNNs are designed to represent two dimensional pure spatial signals but in videos a new temporal axis is added which is essentially different from the spatial variations in images [256, 264].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[172] described available benchmark RGB-D datasets for action/activity recognition and included 27 single-view datasets, 10 multi-view datasets and 7 multi-person datasets.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “With the development of depth sensors, such as Kinect and RealSense, 3D skeleton based human action recognition has received much attention, and a lot of advanced methods have been proposed during the past few years [4], [5], [6], [7].”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “For a more extensive conclusion of activity analysis we refer to corresponding survey papers [5, 56, 1, 7].”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, methodology): “According to the survey of RGB-D datasets [63], we chose two public benchmark datasets, which contain both RGB+depth modalities and have relatively large training samples to evaluate the proposed method.”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, methodology): “With the advent of depth sensors, such as Microsoft Kinect, Asus Xtion, and Intel RealSense, action recognition using 3D skeleton sequences has attracted a lot of research attention, and lots of advanced approaches have been proposed [33, 14, 1, 72].”
Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks (2018, background): “Since the first work of such a type [1] reported in 2010, many methods [2]–[4] have been proposed based on specific hand-crafted feature descriptors extracted from depth.”
Learning features combination for human action recognition from skeleton sequences (2017, background): “The MSR-Action3D [9] is the most common dataset for 3D human action recognition according to [30] and is composed by 10 subjects performing 20 actions chosen in the context of gaming, which include: high wave, horizontal wave, hammer, hand catch, high throw, draw X, draw tick, draw circle, hand…”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, methodology): “After the development of cheap and easy-to-use depth sensors, such as Kinect and XTion, human action recognition in 3D skeleton sequences becomes very popular [1, 79], and a series of hand-crafted features [72, 63, 75, 64, 52, 13, 70, 67] and deep learning based approaches [11, 83, 51, 30, 42, 26] h”
View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition (2018, background): “being robust to appearances, surrounding distractions, and variations of viewpoints [2], [10], [31], [55].”