Space-Time Representation of People Based on 3D Skeletal Data: A Review

By Fei Han, Brian Reily, William A. Hoff, Hao Zhang. 2016

320 citations recorded by the source corpus.

First survey dedicated to human representations based on 3D skeleton data.Our survey is comprehensive and covers the most recent and advanced approaches.An insightful categorization and analysis of the 3D skeleton-based representations is provided. Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Representations can be broadly categorized into two groups, depending on whether they use RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people based on 3D skeletal data, and provides an informative categorization and analysis of these methods from the perspectives, including information modality, representation encoding, structure and transition, and feature engineering. We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of benchmark datasets with skeleton data, and discuss potential future research directions.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “For a more extensive list of current 3D activity analysis datasets and methods, readers are referred to these survey papers [47, 1, 5, 12, 21, 45, 3].”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Readers are referred to these survey papers [31], [32], [33], [34], [35], [36] for a more extensive list of the current 3D activity analysis datasets and methods.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “The sources of the datasets are mainly of three categories (Chen et al., 2013; Cheng et al., 2016; Han et al., 2017): Motion capture (Mocap) system, structured-light cameras (e.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “With the development of depth sensors, such as Kinect and RealSense, 3D skeleton based human action recognition has received much attention, and a lot of advanced methods have been proposed during the past few years [4], [5], [6], [7].”
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, background): “Even without the context information and visual cues, early work [17] in biological perception and more recent methods [9, 10, 31] provide strong evidence that encoding humans as a 3D skeleton yields both a discriminative and a robust representation for activity analysis.”
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data (2016, background): “Skeleton-based human representation has attracted increasing attention for recognizing human actions thanks to its high level representation and robustness to variations of locations and appearances (Han et al. 2016).”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Some survey papers reviewed methods that use the skeleton modality for HAR [35], [36], [37], [38].”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, methodology): “With the advent of depth sensors, such as Microsoft Kinect, Asus Xtion, and Intel RealSense, action recognition using 3D skeleton sequences has attracted a lot of research attention, and lots of advanced approaches have been proposed [33, 14, 1, 72].”
Simultaneous Feature and Body-Part Learning for real-time robot awareness of human behaviors (2017, methodology): “When heterogeneous multimodal features are available, it is well accepted that different types of skeletal features show varying performance on recognizing different behaviors [4].”
Learning Clip Representations for Skeleton-Based 3D Action Recognition (2018, background): “2812099 changes [17], with human actions described as movements of the skeleton joints [18].”