VirtualHome: Simulating Household Activities Via Programs

By Xavier Puig, Kevin Kyunghwan Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba. 2018

691 citations recorded by the source corpus.

In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “training and evaluation tasks, we find that this approach yields 20% more successful policies than other baseline methods in the VirtualHome environment (Puig et al., 2018; 2020).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, methodology): “Recently, many efforts have been put into creating more realistic environments with the goal to further advances in this area (Puig et al., 2018; 4Since the code to train the baseline is not publicly released and a different train/test split is likely used, we only show results reported in Puig et…”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “In VirtualHome [41], programs are generated from video demonstration and natural language instructions, but inference does not involve ego-centric visual and action feedback or partial observability.”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “How are interactive simulators built today? Either by relying on game engines [7, 51, 52] or via a ‘homebrew’ integration of existing rendering and physics libraries [8, 35, 37, 53].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “VirtualHome [21] is a simulator built using the Unity game engine.”
Habitat: A Platform for Embodied AI Research (2019, background): “The availability of large-scale 3D scene datasets [5, 24, 8] and community interest in active vision tasks led to a recent surge of work that resulted in the development of a variety of simulation platforms for indoor environments [17, 7, 13, 22, 26, 3, 27, 28, 21].”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “To allow for more interactive features in environments, researchers leverage partial functionalities of game engines or physics engine to provide photorealistic rendering together with interactions [54, 34, 25, 38, 3, 59, 13, 26].”
The rise and potential of large language model based agents: a survey (2025, neutral): “The constraint is also found in some end-to-end models [254–256].”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “The Household domain is similar to other commonly used benchmarks like ALFWORLD [47] and VirtualHome [40].”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “Interaction in 3D environments Beyond navigation, recent work leverages simulated interactionbased environments [21, 29, 55, 45] to develop agents that can also perform actions (e.”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “, 2022) and vision-based manipulation (Kolve et al., 2017; Puig et al., 2018; Xie et al., 2019; Shridhar et al., 2020a;b; 2022).”
A Survey of Large Language Models (2023, neutral): “VirtualHome [382] builds a 3D simulator for household tasks such as cleaning and cooking, in which the agent can execute natural language actions generated by LLMs.”
PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding (2018, background): “Many cutting-edge learning algorithms, especially for 3D understanding [45, 44, 30, 8], intuitive physics [27], and reinforcement learning [48, 29], require such data to train the networks and benchmark the performances.”
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions (2020, background): “…environments with steadily increasing ﬁdelity are allowing virtual agents to learn to perform high-level tasks that couple language understanding, visual planning, and embodied reasoning through sensorimotor grounded representations (Gordon et al., 2018; Puig et al., 2018; Wijmans et al., 2019).”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, neutral): “Gibson [25] Habitat [18] Sapien [15] AI2Thor [19] VirtualH [20] TDW [26] Provided Large Scenes (RS,AS)”