Ego-Topo: Environment Affordances From Egocentric Video

By Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, Kristen Grauman. 2020

154 citations recorded by the source corpus.

First-person video naturally brings the use of a physical environment to the forefront, since it shows the camera wearer interacting fluidly in a space based on his intentions. However, current methods largely separate the observed actions from the persistent space itself. We introduce a model for environment affordances that is learned directly from egocentric video. The main idea is to gain a human-centric model of a physical space (such as a kitchen) that captures (1) the primary spatial zones of interaction and (2) the likely activities they support. Our approach decomposes a space into a topological map derived from first-person activity, organizing an ego-video into a series of visits to the different zones. Further, we show how to link zones across multiple related environments (e.g., from videos of multiple kitchens) to obtain a consolidated representation of environment functionality. On EPIC-Kitchens and EGTEA+, we demonstrate our approach for learning scene affordances and anticipating future actions in long-form video.

Subfields: Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Other directions of hand-object interaction involve using hands as probes for object understanding [15], affordance hotspot reasoning [47,49], or even leveraging visual hand-object understanding for robotic learning [41,42,48].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18, 19, 21, 40, 62, 67], to predict object information [30, 136] (RGB & 3D) [10], graphs [24] or environment information [28, 81] such as heatmaps [39, 80].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “More closely related to our work, visual affordance models derived from video can detect likely places for actions to occur [38, 18, 40], such as where to grasp a frying pan, and object saliency models can identify human-useable objects in egocentric video [14, 5, 20].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “…of learning a) where and how to grasp [9, 20, 25, 26, 28, 33, 34, 37, 41, 49], b) state classifiers [24], c) interaction hotspots [14, 40, 42, 59], d) spatial priors for action sites [44], e) object articulation modes [11,36], f) reward functions [27,29,48,50], g) functional correspondences [32].”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Examples include detection [11, 15], video recognition [62], interaction hotspots [37], and affordances [38].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “camera wearer location from video inputs [106].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “Some related studies have modeled the relations between entities for interaction recognition as object affordances [43, 44, 22].”
Anticipative Video Transformer (2021, background): “Various approaches have been proposed for this task, such as learning representations by predicting future features [87, 93], aggregating past features [24, 74], or leveraging affordances and hand motion [55, 62].”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “In computer vision, visual affordances are explored in various forms: predicting where to grasp an object from images and video [31, 32, 64, 38, 19, 62, 15, 5], inferring how people might use a space [48, 39] or tool [65], and priors for human body poses [26, 52, 58, 17].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “As shown in [62, 81], the global motion between two consecutive frames is usually small, and they can be related by a homography [77].”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, methodology): “Of these, recent research has explored using EPIC-KITCHENS for: video object reasoning and detection [67], [68], action retrieval [69], visual learning of novel words [57], unsupervised domain adaptation [70] and learning environmental affordances [71].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “Some studies have modeled the relations between entities for interaction recognition as object affordances [65, 66, 25].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Model architectures proposed for these tasks include multi-stream networks [46, 38, 35, 68], recurrent networks [22, 21, 65], 3D conv nets [53, 44] and spatially grounded topological graph models [50].”