Unstructured human activity detection from RGBD images

By Jaeyong Sung, Colin Ponce, Bart Selman, Ashutosh Saxena. 2011

577 citations recorded by the source corpus.

Being able to detect and recognize human activities is essential for several applications, including personal assistive robotics. In this paper, we perform detection and recognition of unstructured human activity in unstructured environments. We use a RGBD sensor (Microsoft Kinect) as the input sensor, and compute a set of features based on human pose and motion, as well as based on image and point-cloud information. Our algorithm is based on a hierarchical maximum entropy Markov model (MEMM), which considers a person's activity as composed of a set of sub-activities. We infer the two-layered graph structure using a dynamic programming approach. We test our algorithm on detecting and recognizing twelve different activities performed by four people in different environments, such as a kitchen, a living room, an office, etc., and achieve good performance even when the person was not seen before in the training set1.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A self-organizing neural network architecture for learning human-object interactions (2017, background): “The most typical approaches are those that do not explicitly model the interplay between object recognition and body pose estimation [27, 28, 29].”
Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks (2016, background): “One is to design robust and discriminative features from the skeleton (and the corresponding RGBD images) for intra-frame content representation (Müller, Röder, and Clausen 2005; Wang, Liu, and Junsong 2012; Sung et al. 2012; Yang and Tian 2014; Ji, Ye, and Cheng 2014).”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “[117] computed the orientation matrix of each human joint with respect to the camera, and then transformed the joint”
3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks (2014, neutral): “7% [25], 80.”
Learning human activities and object affordances from RGB-D videos (2012, methodology): “nds. Human activity detection from RGB-D videos. Recently, with the availability of inexpensive RGB-D sensors, some works (Li et al., 2010; Ni et al., 2011; Sung et al., 2011; Zhang and Parker, 2011; Sung et al., 2012) consider detecting human activities from RGB-D videos. Li et al. (2010) proposed ”
Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, methodology): “We compare the performance of our model with state-ofthe-art methods and baselines on four datasets: MSR Action3D Dataset [26] (Action3D), SBU Interaction dataset [59] (SBU), Cornell Activity Dataset [39](CAD60), and the proposed Blanket dataset (Blanket).”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “Compared to CAD-60, CAD-120 provides extra labels of temporal locations.”
Anticipating Human Activities Using Object Affordances for Reactive Robotic Response (2013, background): “There has been a significant amount of work in detecting human activities from 2D RGB videos [1], [2], [3], from inertial/location sensors [4], and more recently from RGB-D videos [5], [6], [7].”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “As argued in the introduction, direct search does not work, leading to efforts aimed at “acting out” daily activities [39, 23, 35, 11].”
Self-organizing neural integration of pose-motion features for human action recognition (2015, methodology): “In Section 4, we provide experimental results along with an evaluation of our classiﬁcation algorithm on a dataset of 10 full-body actions (Parisi et al., 2014c ) and a benchmark of domestic actions CAD-60 (Sung et al., 2012).”
Discriminative Hierarchical Modeling of Spatio-temporally Composable Human Activities (2014, background): “Other researchers have also addressed the problem of activity recognition on color+depth data, but they usually focus on categorizing non-composed activities [30, 34, 38].”
3D SMoSIFT: three-dimensional sparse motion scale invariant feature transform for activity recognition from RGB-D videos (2014, methodology): “In order to figure up the human pose feature, the authors used the skeleton model, which can move with 15 joints, to calculate 3D coordinate and orientation of each joint.(10) However, the skeleton model is not stable enough to capture 3D positions of tracked joints if serious occlusions occur, whic”
Robobarista: Object Part Based Transfer of Manipulation Trajectories from Crowd-Sourcing in 3D Pointclouds (2015, background): “There has been great advancement in scene understanding [31, 41, 26, 61], in human activity detection [51, 19], and in features for RGB-D images and point-clouds [47, 29].”
Human activities recognition using depth images (2013, unknown): “Copyright 2013 ACM 978-1-4503-2404-5/13/10 ...$15.00.”
Human activity recognition using multi-features and multiple kernel learning (2014, methodology): “[27] compute a set of features based on human pose and motion of the 3D joint positions provided by Prime Sense with Kinect.”
Fusing Spatiotemporal Features and Joints for 3D Action Recognition (2013, background): “Precision/Recall J. Sung et al. [17] 67.9%/55.5% X. Yang et al. [28] 71.9%/66.6% Koppula et al. [18] 80.8%/71.4%”
Structure-Preserving Binary Representations for RGB-D Action Recognition (2016, methodology): “In this section, we systematically evaluate our proposed method on three different RGB-D benchmarks: the SKIG hand gesture dataset [66], the MSRDailyActivity3D dataset [2] and the CAD-60 activity dataset [67].”