Learning About Objects by Learning to Interact with Them

By Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, Roozbeh Mottaghi. 2020

18 citations recorded by the source corpus.

Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks. In contrast, humans often learn about their world with little to no external supervision. Taking inspiration from infants learning from their environment through play and interaction, we present a computational framework to discover objects and learn their physical properties along this paradigm of Learning from Interaction. Our agent, when placed within the near photo-realistic and physics-enabled AI2-THOR environment, interacts with its world and learns about objects, their geometric extents and relative masses, without any external guidance. Our experiments reveal that this agent learns efficiently and effectively; not just for objects it has interacted with before, but also for novel instances from seen categories as well as novel object categories.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Kristen Grauman
2. S. Gadre
3. Cheston Tan
4. Jiafei Duan
5. Samson Yu
6. Tangyao Li
7. Huaiyu Zhu
8. Kiana Ehsani

In Computer Vision and Pattern Recognition

1. Kristen Grauman
2. Tushar Nagarajan
3. Luc Van Gool
4. Jitendra Malik
5. Shuran Song
6. Trevor Darrell
7. Cheston Tan
8. Abhinav Kumar Gupta

Citations in the corpus

• A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “However, due to the nature of game-based scene simulators, many research tasks performed in this environment will opt for its symbolic interaction capabilities as compared to fine-grained object manipulation [3], except for a few that utilize both [2], [130].”
• Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery (2021, methodology): “For learning interaction strategies, Lohmann et al. [26] learn to interact with rigid objects to estimate their segmentation masks and physical properties.”
• AllenAct: A Framework for Embodied AI Research (2020, unknown): “…[18, 19], language grounding [20, 21], grasping [22, 23], object manipulation [24, 25], future prediction [26], and multi-agent collaboration [27, 28]; as well as using interaction as a tool to learn: environment representations [29], intuitive physics [30], and objects and attributes [31].”
• Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Recent work learns visual priors from agent experience (not human video) to understand environment traversability [46], affordances [39], or object properties like mass [36].”