Synthesizing Environment-Aware Activities via Activity Sketches
By Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, Sanja Fidler. 2019
44 citations recorded by the source corpus.
In order to learn to perform activities from demonstrations or descriptions, agents need to distill what the essence of the given activity is, and how it can be adapted to new environments. In this work, we address the problem: environment-aware program generation. Given a visual demonstration or a description of an activity, we generate program sketches representing the essential instructions and propose a model to flesh these into full programs representing the actions needed to perform the activity under the presented environmental constraints. To this end, we build upon VirtualHome, to create a new dataset VirtualHome-Env, where we collect program sketches to represent activities and match programs with environments that can afford them. Furthermore, we construct a knowledge base to sample realistic environments and another knowledge base to seek out the programs under the sampled environments. Finally, we propose RNN-ResActGraph, a network that generates a program from a given sketch and an environment graph and tracks the changes in the environment induced by the program.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Josh Tenenbaum
- Igor Mordatch
- Antonio Torralba
- Pieter Abbeel
- Chelsea Finn
- Abhinav Kumar Gupta
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Antonio Torralba
- Dacheng Tao
- Ming-Hsuan Yang
- Andrea Vedaldi
In Information Systems
- Antonio Torralba
- Pieter Abbeel
- Josh Tenenbaum
- Tianmin Shu
- Shao-Hua Sun
- Joseph J. Lim
- Sanja Fidler
- Lise Getoor
Citations in the corpus
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “However, relevant works (Puig et al., 2018; Liao et al., 2019) rely on human-annotated data and perform supervised training from scratch.”
- Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “Building upon Liao et al. (2019), we represent the symbolic observations as a state graph with each node representing the class label and physical state of an object, and each edge representing the spatial relation of two objects.”
- Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[42] Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler.”