3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

By Keze Wang, X. Wang, Liang Lin, M. Wang, Wangmeng Zuo. 2014

123 citations recorded by the source corpus.

Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

RGB-D-based action recognition datasets: A survey (2016, methodology): “Cluttered background and occlusion: UTKinect, RGBD-HuDaAct, MSRDaily Activity, CAD-120, RGBD-SAR, 3D Online, DMLSmartActions, Multiview 3D Event, LIRIS, and Oﬃce Activity.”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, neutral): “MSR-Action3D [17] 20 567 567 1 D+Skeleton No 2010 RGBD-HuDaAct [21] 13 1189 1189 1 RGB+D No 2011 MSR-DailyActivity [43] 16 320 320 1 RGB+D+Skeleton No 2012 Act4 [8] 14 6844 6844 1 RGB+D No 2012 UTKinect-Action [53] 10 200 200 1 RGB+D+Skeleton No 2012 3D Action Pairs [23] 12 360 360 1 RGB+D+Skeleton ”
A Deep Structured Model with Radius–Margin Bound for 3D Human Activity Recognition (2015, methodology): “First, the conﬁguration of our deep model can be ﬂexi-bly adjusted to adapt to different input videos, and the sig-niﬁcance of this property has been justiﬁed for human action recognition [22, 38, 45].”
Cracking Classifiers for Evasion: A Case Study on the Google's Phishing Pages Filter (2016, background): “The amount of weight connections are millions, as exhibited in [55].”
Human Action Recognition and Prediction: A Survey (2018, background): “Action features learned by deep learning techniques has been popularly investigated [18], [30], [31], [154], [155], [156], [157], [158], [159], [160], [161], [162] in recent years.”
Human action recognition using genetic algorithms and convolutional neural networks (2016, methodology): “proposed a deep learning model for human activity recognition in [27] by extending a CNN to incorporate structure alternatives by using latent variables in convolutional layers to manipulate the activation of neurons.”