First Person Action-Object Detection with EgoNet

By Gedas Bertasius, Hyun Soo Park, Stella X. Yu, Jianbo Shi. 2016

61 citations recorded by the source corpus.

Unlike traditional third-person cameras mounted on robots, a first-person camera, captures a person's visual sensorimotor object interactions from up close. In this paper, we study the tight interplay between our momentary visual attention and motor action with objects from a first-person camera. We propose a concept of action-objects---the objects that capture person's conscious visual (watching a TV) or tactile (taking a cup) interactions. Action-objects may be task-dependent but since many tasks share common person-object spatial configurations, action-objects exhibit a characteristic 3D spatial distance and orientation with respect to the person. We design a predictive model that detects action-objects using EgoNet, a joint two-stream network that holistically integrates visual appearance (RGB) and 3D spatial layout (depth and height) cues to predict per-pixel likelihood of action-objects. Our network also incorporates a first-person coordinate embedding, which is designed to learn a spatial distribution of the action-objects in the first-person data. We demonstrate EgoNet's predictive power, by showing that it consistently outperforms previous baseline approaches. Furthermore, EgoNet also exhibits a strong generalization ability, i.e., it predicts semantically meaningful objects in novel first-person datasets. Our method's ability to effectively detect action-objects could be used to improve robots' understanding of human-object interactions.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “The authors of [3] detected the important objects for the camera wearer (i.”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “Understanding the interactions between a camera wearer and the surrounding objects is a fundamental problem in First Person Vision (FPV) [19, 81, 51, 30, 20, 68, 12, 5, 6].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18–20, 39, 59, 63], to predict object information [29, 129] (RGB & 3D) [10], graphs [23] or environment information [27, 76] such as heatmaps [38, 75].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “More closely related to our work, visual affordance models derived from video can detect likely places for actions to occur [38, 18, 40], such as where to grasp a frying pan, and object saliency models can identify human-useable objects in egocentric video [14, 5, 20].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “…have considered the problems of detecting action-objects (i.e., objects which will not necessary be involved in future actions, but attract the camera wearer’s attention) [60, 61] and predicting the location at which objects and hands will appear in future frames (e.g., 1 or 5 seconds later) [62].”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, methodology): “In [10], a two-stream visual segmentation-based architecture is used to predict the interaction areas between hands and objects in a video stream and model them as actions.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “It is also common to extract features with CNNs and leverage additional 2D cues related to motion, hand location, object location or object class, in isolation [3, 46, 53] or jointly [19, 21, 25, 67, 74].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, unknown): “EgoNet [2] detects “action objects”, i.”