Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

By Hema Swetha Koppula, Ashutosh Saxena. 2013

769 citations recorded by the source corpus.

An important aspect of human perception is anticipation, which we use extensively in our day-to-day activities when interacting with other humans as well as with our surroundings. Anticipating which activities will a human do next (and how) can enable an assistive robot to plan ahead for reactive responses. Furthermore, anticipation can even improve the detection accuracy of past activities. The challenge, however, is two-fold: We need to capture the rich context for modeling the activities and object affordances, and we need to anticipate the distribution over a large space of future human activities. In this work, we represent each possible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations through object affordances. We then consider each ATCRF as a particle and represent the distribution over the potential futures using a set of particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, we first show that anticipation improves the state-of-the-art detection results. We then show that for new subjects (not seen in the training set), we obtain an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of 84.1, 74.4 and 62.2 percent for an anticipation time of 1, 3 and 10 seconds respectively. Finally, we also show a robot using our algorithm for performing a few reactive responses.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Social Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Social Psychology

In Reinforcement Learning

Citations in the corpus

Generating Videos with Scene Dynamics (2016, background): “Our work is also related to efforts to predict the future in video [33, 22, 44, 51, 42, 17, 8, 55] as well as concurrent work in future generation [6, 15, 20, 50, 56, 43].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, neutral): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Top-k accounts for uncertainty in future predictions, as with previous anticipation efforts [83,84,85].”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, methodology): “Finally, supervised learning has also been used to model and forecast labeled human-human [2,16], and human-object [17,18] interactions.”
Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015, methodology): “Several works follow a two-stage approach and decouple the deep network from CRF.”
Dense Optical Flow Prediction from a Static Image (2015, neutral): “This includes forecasting pedestrian trajectories [10], human-human interactions [7, 15], human expressions through SOSVM [5], and human-object interaction through graphical models [11, 3].”
Inferring the Why in Images (2014, background): “Action Prediction: There have been several works in robotics that predicts a person’s imminent next action from a sequence of images [31, 23, 15, 8, 17].”
Anticipating the future by watching unlabeled video (2015, background): “For example, to make plans or interact with humans, robots will require predictions of the future [19].”
SceneGrok (2014, background): “…not new; much prior work has targeted this domain [Gibson 1977; Gupta et al. 2011; Pandey and Alami 2012; Fouhey et al. 2012; Koppula et al. 2013; Koppula and Saxena 2013; ACM Transactions on Graphics, Vol. 33, No. 6, Article 212, Publication Date: November 2014 Jiang et al. 2013; Wei et al.…”
A Hierarchical Representation for Future Action Prediction (2014, background): “For example, Koppula and Saxena [9] address the problem of anticipating future activities from RGB-D data by considering human-object interactions.”
PiGraphs (2016, background): “The seminal work on affordances by Gibson [1977] has provided inspiration for leveraging human-object interaction data in a variety of tasks, including the improvement of pose estimation, object recognition, action classification, and many other problems [Stark et al. 2008; Bohg et al. 2013; Koppula”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, background): “In [7], observed activities are modeled with spatio-temporal graphs which are used for anticipating object affordances, trajectories, and sub-activities.”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Prior works in immediate anticipation were initially limited to movement primitives like reaching [17] or interactions such as hugging [38].”
Peeking Into the Future: Predicting Future Person Activities and Locations in Videos (2019, background): “Another line of works is anticipating human activities in robotic vision [15, 10].”
Social LSTM: Human Trajectory Prediction in Crowded Spaces (2016, background): “Pioneering works by [24, 50, 35] have also proposed ways to model human-human interactions (often called ”social forces”) to increase robustness and accuracy in multi-target tracking problems.”