Learning human activities and object affordances from RGB-D videos

By Hema Swetha Koppula, Rudhir Gupta, Ashutosh Saxena. 2012

756 citations recorded by the source corpus.

Understanding human activities and object affordances are two very important skills, especially for personal robots which operate in human environments. In this work, we consider the problem of extracting a descriptive labeling of the sequence of sub-activities being performed by a human, and more importantly, of their interactions with the objects in the form of associated affordances. Given a RGB-D video, we jointly model the human activities and object affordances as a Markov random field where the nodes represent objects and sub-activities, and the edges represent the relationships between object affordances, their relations with sub-activities, and their evolution over time. We formulate the learning problem using a structural support vector machine (SSVM) approach, where labelings over various alternate temporal segmentations are considered as latent variables. We tested our method on a challenging dataset comprising 120 activity videos collected from 4 subjects, and obtained an accuracy of 79.4% for affordance, 63.4% for sub-activity and 75.0% for high-level activity labeling. We then demonstrate the use of such descriptive labeling in performing assistive tasks by a PR2 robot.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “CAD-60 [34] and CAD-120 [18] contain RGB, depth, and skeleton data of human actions.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “The CAD-60 [12] and CAD-120 [17] datasets contain RGB, depth, and skeleton data of human actions.”
A self-organizing neural network architecture for learning human-object interactions (2017, methodology): “In Table 2, we show a comparison of our results with the state of the art on the CAD-120 dataset with accuracy, precision, and re-call as evaluation metrics.”
RGB-D datasets using microsoft kinect or similar sensors: a survey (2016, methodology): “The curves under the images are the accelerometer data at 50 Hz of devices attached to the knife, the mixing spoon, the small spoon, the peeler, the glass, the oil bottle, and the pepper dispenser mentioned datasets above, another 26 extra RGB-D datasets for different applications are also added int”
RGB-D-based action recognition datasets: A survey (2016, background): “Human–object interactivity is another important characteristic of a dataset because some algorithms may benefit from the objects that the actors interact with [60,61,22].”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “6M [92] MoCap color movies 2014 Northwestern-UCLA Multiview [106] Kinect v1 RGB + depth human daily activities 2014 ORGBD [107] Kinect v1 RGB + depth human-object interactions 2014 SPHERE [108] Kinect depth human daily activities 2014 TST Fall Detection [109] Kinect v2 + IMU depth human daily activi”
3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks (2014, methodology): “We validate our approach on the CAD-120 activity dataset [9] and a larger dataset newly created by us, namely Oﬃce Activity (OA) dataset.”
Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “[36] which contains 120 different RGB-D camera sequences where four individuals perform activities like making cereals, microwaving food and more.”
Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015, methodology): “The applications include: (i) modeling human motion [14] from motion capture data [21]; (ii) human activity detection and anticipation [29, 31]; and (iii) maneuver anticipation from real-world driving data [22].”
Anticipating Human Activities Using Object Affordances for Reactive Robotic Response (2013, methodology): “( KGS ' s method was to label a segment given past , present , as well as the future .”
Hallucinated Humans as the Hidden Context for Labeling 3D Scenes (2013, background): “Only recently, some works [9, 32, 1, 25, 20] have shown that modeling the interaction between human poses and objects in 2D images and videos result in a better performance on the tasks of object detection and activity recognition.”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, background): “Pioneer work [28], [34] on using object keypoints for robot manipulation has mostly focused on training from supervised learning on datasets with manual annotations.”
A survey of robot learning from demonstrations for Human-Robot Collaboration (2017, background): “The authors demonstrated the proposed approach on the CAD-120 human activity dataset [26] and obtained 2.7% improvement on the state-of-the-art detection results in terms of a success rate.”
Dynamic Graph Convolutional Networks (2017, background): “CAD-120 6 is a dataset composed of 122 RGB-D videos corresponding to 10 high-level human activities [13].”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, methodology): “Our set of object affordance is inspired by previous work (Kjellström et al., 2011; Koppula et al., 2013; Montesano et al., 2008).”