Building Generalizable Agents with a Realistic and Rich 3D Environment

By Yi Wu, Yuxin Wu, Georgia Gkioxari, Yuandong Tian. 2018

361 citations recorded by the source corpus.

Towards bridging the gap between machine and human intelligence, it is of utmost importance to introduce environments that are visually realistic and rich in content. In such environments, one can evaluate and improve a crucial property of practical intelligent systems, namely \emph{generalization}. In this work, we build \emph{House3D}, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of houses, ranging from single-room studios to multi-storeyed houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et al., 2017). With an emphasis on semantic-level generalization, we study the task of concept-driven navigation, \emph{RoomNav}, using a subset of houses in House3D. In RoomNav, an agent navigates towards a target specified by a semantic concept. To succeed, the agent learns to comprehend the scene it lives in by developing perception, understand the concept by mapping it to the correct semantics, and navigate to the target by obeying the underlying physical rules. We train RL agents with both continuous and discrete action spaces and show their ability to generalize in new unseen environments. In particular, we observe that (1) training is substantially harder on large house sets but results in better generalization, (2) using semantic signals (e.g., segmentation mask) boosts the generalization performance, and (3) gated networks on semantic input signal lead to improved training performance and generalization. We hope House3D, including the analysis of the RoomNav task, serves as a building block towards designing practical intelligent systems and we wish it to be broadly adopted by the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “This opens up new possibilities for training language-aware agents: in the real world, and even in rich simulated environments (Brodeur et al., 2017; Wu et al., 2018), acquiring such data via human annotation would often be much more viable than deﬁning and implementing reward functions…”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “Some simulators [3, 49, 50] involve largely static environments – the agent can move but not change the state of the environment (e.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “sting 3D RL environments based on game engines, such as ViZDoom [27], DeepMind Lab [7] and AI2-THOR [30], as well as a number of newer environments developed concurrently including HoME [10], House3D [58], MINOS [47], CHALET [59] and Gibson Env [61]. The main advantage of our framework over syntheti”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “Nonphotorealistic datasets and environments [29, 30, 70, 78] also play an important role in scene understanding research because they can be rendered very efficiently.”
Embodied Question Answering (2017, methodology): “In the descriptions that follow, an ‘entity’ can refer to either a queryable room or a queryable object from the House3D [8] environment.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “…AI2-Thor (Kolve et al., 2017) Color 100s+FPS Indoor Navigation Minos (Savva et al., 2017) Depth/Color/Semantics 100s+FPS Indoor Navigation House3D (Wu et al., 2018) Depth/Color/Semantics 600s+FPS Indoor Navigation Independent VS. Compounding Errors : Standard supervised learning assumes that the…”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “For example, in navigation games (Mirowski et al., 2016; Lowe et al., 2017; Wu et al., 2018), the reward is typically set to the negative distance from the target location L T to the agent’s location L A plus a success bonus, so the feature vector φ ( s, a ) can be written as a 2-dimensional vector…”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “Researchers have also gone beyond specifying goals as a desired location in space and finding objects of interest as done in this paper, for example, Wu et al. (2018) generalize the goal specification to also include rooms of interest, and Das et al. (2018) allow goal specification via templated…”
Habitat: A Platform for Embodied AI Research (2019, background): “The availability of large-scale 3D scene datasets [5, 24, 8] and community interest in active vision tasks led to a recent surge of work that resulted in the development of a variety of simulation platforms for indoor environments [17, 7, 13, 22, 26, 3, 27, 28, 21].”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, methodology): “The IR baseline shows non-random performance, but is outperformed by Starspace which is a stronger baseline.”
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “Since then, a number of independent efforts have investigated navigation in indoor environments using the SUNCG dataset (Wu et al., 2017; Brodeur et al., 2017; Das et al., 2017) and the Matterport3D dataset (Anderson et al., 2017).”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “First, BabyAI supports world state manipulation, missing in the visually appealing 3D environments of Hermann et al. (2017), Chaplot et al. (2018) and Wu et al. (2018).”
On Evaluation of Embodied Navigation Agents (2018, background): “A number of simulation platforms for indoor environments have recently been proposed [4, 12, 16, 28, 30, 31].”
VirtualHome: Simulating Household Activities Via Programs (2018, background): “Recent works have focused on simulating indoor environments, allowing for target-driven indoor navigation or interactive question answering [11, 28, 6, 22].”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, methodology): “We also apply reward shaping (Wu et al., 2018): the direct reward at each non-stop step t is the change of the distance to the target viewpoint.”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “Virtual environments have been a remedy for this, classically [64] and today [100, 34, 29, 79, 44, 10, 8, 68, 92].”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “In recent years, there has been a proliferation of indoor simulation environments primarily designed for navigation, visual recognition and reasoning [43, 56, 58, 44, 1].”
Neural Modular Control for Embodied Question Answering (2018, methodology): “convolutional neural network trained to perform auto-encoding, semantic segmentation, and depth estimation from RGB frames taken from House3D [2].”