Grounded Human-Object Interaction Hotspots From Video

By Tushar Nagarajan, Christoph Feichtenhofer, Kristen Grauman. 2018

202 citations recorded by the source corpus.

Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating where an object would be manipulated in a potential interaction even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories. Project page: http://vision.cs.utexas.edu/projects/interaction-hotspots/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Other directions of hand-object interaction involve using hands as probes for object understanding [15], affordance hotspot reasoning [47,49], or even leveraging visual hand-object understanding for robotic learning [41,42,48].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, neutral): “[163] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman.”
Understanding Human Hands in Contact at Internet Scale (2020, background): “Most existing work in this area that can learn from Internet videos [27, 13] focuses on interaction hotspots, while our work infers a mesh.”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “Some studies have modeled the relations between entities for interaction recognition as object affordances [65, 66, 25].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “Some related studies have modeled the relations between entities for interaction recognition as object affordances [42, 43, 21].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, methodology): “VRB consistently outperforms our most competitive baseline, Hotspots [80].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Hotspots [61] anticipated spatial interaction regions using Grad-Cam [71], given the future action label as additional input.”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, methodology): “In computer vision, visual affordances are explored in various forms: predicting where to grasp an object from images and video [31, 32, 64, 38, 19, 62, 15, 5], inferring how people might use a space [48, 39] or tool [65], and priors for human body poses [26, 52, 58, 17].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “This special viewpoint offers an array of interesting challenges, such as detecting gaze [40, 29], monitoring human-object interactions [4, 6, 51], creating daily life activity summaries [44, 39, 70, 43], or inferring the camera wearer’s identity or body pose [28, 33].”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Examples include detection [11, 15], video recognition [62], interaction hotspots [37], and affordances [38].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “[42] that derives supervision from manually annotated object bounding boxes and action labels in the EPIC-KITCHENS dataset.”
DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (2022, background): “Vision methods explore learning affordances from visual data [36, 37, 38], though they stop short of agent action.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “In [64, 112, 119], spatial attention has been used to predict the future gaze, while in [50, 63], it has been used as a mechanism to solve the task of predicting interaction hotspots.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “More closely related to our work, visual affordance models derived from video can detect likely places for actions to occur [38, 18, 40], such as where to grasp a frying pan, and object saliency models can identify human-useable objects in egocentric video [14, 5, 20].”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “Visual affordances A few methods infer visual affordances for grasping with simple grippers [40], [19], [23], [20] and explore non-robotics affordances [32], [9], [34], [11].”