Learning Affordance Landscapes for Interaction Exploration in 3D Environments

By Tushar Nagarajan, Kristen Grauman. 2020

85 citations recorded by the source corpus.

Embodied agents operating in human spaces must be able to master how their environment works: what objects can the agent use, and how can it use them? We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen). Given an egocentric RGB-D camera and a high-level action space, the agent is rewarded for maximizing successful interactions while simultaneously training an image-based affordance segmentation model. The former yields a policy for acting efficiently in new environments to prepare for downstream interaction tasks, while the latter yields a convolutional neural network that maps image regions to the likelihood they permit each action, densifying the rewards for exploration. We demonstrate our idea with AI2-iTHOR. The results show agents can learn how to use new home environments intelligently and that it prepares them to rapidly address various downstream tasks like "find a knife and put it in the drawer." Project page: this http URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “For physics, the furtherance of physics-based predictive models [113]–[115] have accentuate on the importance of embodied AI simulators with advanced physics features as they serve to provide an ideal testbed for training embodied AI agents to perform tasks with sophisticated physical interactions [”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Embodied AI (EAI) agents take in multi-sensory input, such as vision and language [22] or vision and audio [16] and take actions such as navigation [3], object manipulation [48, 51], or both [30, 66], in a simulated environment.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, methodology): “ScenePriors [65] NavExp [49] IntExp [39] Ours Fig.”
Language Grounding with 3D Objects (2021, background): “Agents can perform information-seeking behaviors on objects to improve their world [15, 16, 17] and language understanding [18, 19, 20].”