MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments

By Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas A. Funkhouser, Vladlen Koltun. 2017

259 citations recorded by the source corpus.

We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. The simulator leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites. We use MINOS to benchmark deep-learning-based navigation methods, to analyze the influence of environmental complexity on navigation performance, and to carry out a controlled study of multimodality in sensorimotor learning. The experiments show that current deep reinforcement learning approaches fail in large realistic environments. The experiments also indicate that multimodality is beneficial in learning to navigate cluttered scenes. MINOS is released open-source to the research community at this http URL . A video that shows MINOS can be found at this https URL

Subfields: Cognitive Neuroscience, Computer Science Applications, Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “Some simulators [3, 49, 50] involve largely static environments – the agent can move but not change the state of the environment (e.”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “Learning to navigate in a photorealistic home environment based on natural language instructions has attracted increasing research interest [23, 14, 7, 3, 6], as it provides insight into core scientiﬁc questions about multimodal representations.”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “SceneNN [31] X S+I Stanford 2D-3D-S [8, 9, 75] X X S+I Matterport3D [6, 14, 57, 58, 75] X X S+I ScanNet [17] X X S+I Gibson [58, 75] X”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “…Driving Torcs (You et al., 2017) Color/Semantics 100s+FPS Autonomous Driving AI2-Thor (Kolve et al., 2017) Color 100s+FPS Indoor Navigation Minos (Savva et al., 2017) Depth/Color/Semantics 100s+FPS Indoor Navigation House3D (Wu et al., 2018) Depth/Color/Semantics 600s+FPS Indoor Navigation…”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39, 40, 41, 42, 43], autonomous driving [44, 45, 46], grasping [47, 48, 49], single-task manipulation [50], among others.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “Numerous works such as (Chang et al. 2017; Dai et al. 2017; Ammirato et al. 2017) have collected large-scale datasets consisting of scans of real world environments, while (Savva et al. 2017; Wu et al. 2018; Xia et al. 2018) have built more sophisticated simulation environments based on such scans.”
Habitat: A Platform for Embodied AI Research (2019, methodology): “By comparison, AI2-THOR [17] and CHALET [28] run at tens of fps, MINOS [22] and Gibson [27] run at about a hundred, and House3D [26] runs at about 300 fps. Habitat-Sim is 2-3 orders of magnitude faster.”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, methodology): “A number of visual, rather than text, platforms have been proposed such as House3D (Wu et al., 2018b), HoME (Brodeur et al., 2017), MINOS (Savva et al., 2017), Matterport3D (Chang et al., 2017) and AI2-THOR (Kolve et al., 2017), and the Minecraft MALMO project (Johnson et al., 2016), but they typica”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Two concurrent works (Brodeur et al., 2017; Savva et al., 2017) introduce a very similar platform to House3D, indicating the interest for large-scale interactive and realistic 3D environments.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “In the past two years, a plethora of tasks and evaluation protocols [32, 22, 34, 45, 2] have been proposed as summarized in [1].”
On Evaluation of Embodied Navigation Agents (2018, background): “A number of simulation platforms for indoor environments have recently been proposed [4, 12, 16, 28, 30, 31].”
Semi-parametric Topological Memory for Navigation (2018, methodology): “DL-based approaches to navigation can be broadly divided into three classes: purely reactive (Dosovitskiy & Koltun, 2017; Zhu et al., 2017), based on unstructured general-purpose memory such as LSTM (Mnih et al., 2016; Mirowski et al., 2017), and employing a navigation-speciﬁc memory structure…”
VirtualHome: Simulating Household Activities Via Programs (2018, background): “Recent works have focused on simulating indoor environments, allowing for target-driven indoor navigation or interactive question answering [11, 28, 6, 22].”
Gibson Env: Real-World Perception for Embodied Agents (2018, methodology): “§ Rendered with Gibson, † rendered with MINOS [72].”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “In recent years, there has been a proliferation of indoor simulation environments primarily designed for navigation, visual recognition and reasoning [43, 56, 58, 44, 1].”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, background): “This behavior is prevalent in video game engines as it allows for smooth human control; it is also enabled by default in MINOS [22], Deepmind Lab [13], AI2 THOR [19], and Gib-son [17] v1.”
Multi-Target Embodied Question Answering (2019, background): “In the vision, language, and AI communities, we are witnessing a shift in focus from internet vision to embodied AI – with the creation of new tasks and benchmarks [7, 2, 14, 35], instantiated on new simulation platforms [21, 28, 32, 33, 20, 6].”
Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments (2019, background): “[39] M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun, “MINOS: Multimodal indoor simulator for navigation in complex environments,” 2017, arXiv:1712.03931.”