HoME: a Household Multimodal Environment

By Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, L. Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, Aaron C. Courville. 2017

107 citations recorded by the source corpus.

We introduce HoME: a Household Multimodal Environment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context. HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning, and more. We hope HoME better enables artificial agents to learn as humans do: in an interactive, multimodal, and richly contextualized setting.

Subfields: Artificial Intelligence, Social Psychology, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Each module m x performs a convolution (weights shared by all modules) followed by a token-speciﬁc Feature-Wise Linear Modulation (FiLM) (Perez et al., 2017) where h l and h r are module inputs, γ x is a vector of FiLM multipliers, β x are FiLM biases, (cid:12) and ⊕ are element-wise multiplication…”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “related to existing 3D RL environments based on game engines, such as ViZDoom [27], DeepMind Lab [7] and AI2-THOR [30], as well as a number of newer environments developed concurrently including HoME [10], House3D [58], MINOS [47], CHALET [59] and Gibson Env [61]. The main advantage of our framework”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39, 40, 41, 42, 43], autonomous driving [44, 45, 46], grasping [47, 48, 49], single-task manipulation [50], among others.”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “DeepMind Lab [3] X X VizDoom [13] X X MINOS (Matterport3D) [17] X X MINOS (SUNCG) [17] X X X House3D [2] X X X HoME [5] X X X X AI2-THOR X X X X X Table 1.”
Habitat: A Platform for Embodied AI Research (2019, background): “ 3D scene datasets [5, 27, 8] and community interest in active vision tasks led to a recent surge of work that resulted in the development of a variety of simulation platforms for indoor environments [17, 7, 13, 24, 29, 3, 30, 31, 23]. These platforms vary with respect to the 3D scene data they use,”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, methodology): “A number of visual, rather than text, platforms have been proposed such as House3D (Wu et al., 2018b), HoME (Brodeur et al., 2017), MINOS (Savva et al., 2017), Matterport3D (Chang et al., 2017) and AI2-THOR (Kolve et al., 2017), and the Minecraft MALMO project (Johnson et al., 2016), but they typica”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Two concurrent works (Brodeur et al., 2017; Savva et al., 2017) introduce a very similar platform to House3D, indicating the interest for large-scale interactive and realistic 3D environments.”
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “Since then, a number of independent efforts have investigated navigation in indoor environments using the SUNCG dataset (Wu et al., 2017; Brodeur et al., 2017; Das et al., 2017) and the Matterport3D dataset (Anderson et al., 2017).”
On Evaluation of Embodied Navigation Agents (2018, background): “A number of simulation platforms for indoor environments have recently been proposed [4, 12, 16, 28, 30, 31].”
VirtualHome: Simulating Household Activities Via Programs (2018, background): “A few of these works [11, 6] include actionable objects, allowing to interact and change the environment.”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “Simulation environments that leverage game engines [52, 33, 3, 13, 25] come with manually designed or procedurally generated game scenes.”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, unknown): “To reach this goal, we have seen the increasing use of synthetic data and simulation platforms for indoor scenes in recent years by leveraging state-of-the-art game engines and free, publicly available 3D content [288,375–377], including MINOS [378], HoME [379], Gibson Environment [380], House3D [38”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “, 2018) Low Ego Yes No No No HoME (Brodeur et al., 2017) Low Ego Yes No No No Gibson (Xia et al.”
The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation (2019, background): “Building on the success of deep learning in solving various computer vision tasks, several new tasks and corresponding benchmarks have been proposed to combine visual perception and decision-making [2, 22, 7, 12, 17, 23, 5].”
Audio-Visual Embodied Navigation (2019, background): “While the HoME simulator [11] report states support for acoustics, there are no instructions for its use and the environment is no longer actively maintained.”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “Instruction following have recently drawn a lot of attention following the emergence of several 2D and 3D environments [12, 8, 1].”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “In recent years, several synthetic frameworks have been proposed to investigate tasks including visual navigation, task completion and question answering in indoor scenes [32, 53, 70, 8, 74, 72, 46, 54].”
Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction (2019, unknown): “Significant progress has been made in many real-world applications, such as playing video games [17, 16, 4], robot control [6, 14], and robot navigation [12, 5, 32, 27].”
Why Build an Assistant in Minecraft? (2019, background): “[16] Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville.”
Multi-Target Embodied Question Answering (2019, background): “In the vision, language, and AI communities, we are witnessing a shift in focus from internet vision to embodied AI – with the creation of new tasks and benchmarks [7, 2, 14, 35], instantiated on new simulation platforms [21, 28, 32, 33, 20, 6].”