Action Recognition Using Rate-Invariant Analysis of Skeletal Shape Trajectories

By Boulbaba Ben Amor, Jingyong Su, Anuj Srivastava. 2016

215 citations recorded by the source corpus.

We study the problem of classifying actions of human subjects using depth movies generated by Kinect or other depth sensors. Representing human body as dynamical skeletons, we study the evolution of their (skeletons') shapes as trajectories on Kendall's shape manifold. The action data is typically corrupted by large variability in execution rates within and across subjects and, thus, causing major problems in statistical analyses. To address that issue, we adopt a recently-developed framework of Su et al. [1], [2] to this problem domain. Here, the variable execution rates correspond to re-parameterizations of trajectories, and one uses a parameterization-invariant metric for aligning, comparing, averaging, and modeling trajectories. This is based on a combination of transported square-root vector fields (TSRVFs) of trajectories and the standard Euclidean norm, that allows computational efficiency. We develop a comprehensive suite of computational tools for this application domain: smoothing and denoising skeleton trajectories using median filtering, up- and down-sampling actions in time domain, simultaneous temporal-registration of multiple actions, and extracting invertible Euclidean representations of actions. Due to invertibility these Euclidean representations allow both discriminative and generative models for statistical analysis. For instance, they can be used in a SVM-based classification of original actions, as demonstrated here using MSR Action-3D, MSR Daily Activity and 3D Action Pairs datasets. Using only the skeletal information, we achieve state-of-the-art classification results on these datasets.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “[173] proposed to model the evolution of human skeleton shapes as trajectories on Kendall’s shape manifolds, and used a parameterization-invariant metric [217] for aligning, comparing, and modeling skeleton joint trajectories, which can deal with noise caused by large variability of execution rates ”
Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions (2017, background): “Such representations include local spatio-temporal interest points and features [26, 31, 52] and local trajectories [1, 50], typically aggregated into VLAD or Fisher vector representations [38, 39].”
A Novel Space-Time Representation on the Positive Semidefinite Cone for Facial Expression Recognition (2017, background): “Recently, shape trajectory-based methods showed their effectiveness in many temporal pattern recognition tasks, especially in action recognition [2, 4, 6, 9, 40].”
Coding Kendall's Shape Trajectories for 3D Action Recognition (2018, methodology): “Moreover, as in [2] human actions are also first represented as trajectories in the Kendall’s shape space, we report additional results of [2] on Florence3D and UTKinect datasets to give more insights about the strength of our coding approach compared to the method of [2].”
Simultaneous Feature and Body-Part Learning for real-time robot awareness of human behaviors (2017, background): “…al. [26] Action Graph 74.7% Xia et al. [28] HOJ3D 78% Yang and Tian [9] EigenJoints 83.3% Wang et al. [2] Actionlet Ensemble 88.2% Ben Amor et al. [29] Skeleton Trajectories 89% Feature Learning Only 85.00% Our Methods Body-Part Learning Only 86.67% FABL 91.67% with two males and two females (one…”