Human Hands as Probes for Interactive Object Understanding
By Mohit Goyal, Sahil Modi, Rishabh Goyal, Saurabh Gupta. 2021
66 citations recorded by the source corpus.
Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features, and object affordances (regions of interaction and afforded grasps), purely by observing hands in egocentric videos.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Jitendra Malik
- Luc Van Gool
- Trevor Darrell
- Abhinav Kumar Gupta
- Li Fei-Fei
- Kristen Grauman
- Dacheng Tao
- Andrew Zisserman
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Suraj Nair
- Vikash Kumar
- Aravind Rajeswaran
- Pieter Abbeel
- Russell Mendonca
- Shikhar Bahl
In Human-Computer Interaction
- Dandan Shan
- Lingzhi Zhang
- Dima Damen
- David F. Fouhey
- Aravind Rajeswaran
- Vikash Kumar
- Shenghao Zhou
- Simon Stent
Citations in the corpus
- EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “Most work with segments has focused only on hands [3] or has been gathered at the few thousand scale [14, 36].”
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Other directions of hand-object interaction involve using hands as probes for object understanding [15], affordance hotspot reasoning [47,49], or even leveraging visual hand-object understanding for robotic learning [41,42,48].”
- Affordances from Human Videos as a Versatile Representation for Robotics (2023, methodology): “More general interaction information can be learned from large human datasets [18, 19, 21, 40, 62, 67], to predict object information [30, 136] (RGB & 3D) [10], graphs [24] or environment information [28,81] such as heatmaps [39, 80].”
- R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Finally, there is a rich literature of works that study learning image representations from videos [60, 61, 19, 62, 63, 64] outside of the context of robotics.”