Are We Making Real Progress in Simulated Environments? Measuring the Sim2Real Gap in Embodied Visual Navigation

By Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, Dhruv Batra. 2019

56 citations recorded by the source corpus.

Does progress in simulation translate to progress in robotics? Specifically, if method A outperforms method B in simulation, how likely is the trend to hold in reality on a robot? We examine this question for embodied (PointGoal) navigation, developing engineering tools and a research paradigm for evaluating a simulator by its sim2real predictivity, revealing surprising findings about prior work. First, we develop Habitat-PyRobot Bridge (HaPy), a library for seamless execution of identical code on a simulated agent and a physical robot. Habitat-to-Locobot transfer with HaPy involves just one line change in config, essentially treating reality as just another simulator! Second, we investigate sim2real predictivity of Habitat-Sim for PointGoal navigation. We 3D-scan a physical lab space to create a virtualized replica, and run parallel tests of 9 different models in reality and simulation. We present a new metric called Sim-vs-Real Correlation Coefficient (SRCC) to quantify sim2real predictivity. Our analysis reveals several important findings. We find that SRCC for Habitat as used for the CVPR19 challenge is low (0.18 for the success metric), which suggests that performance improvements for this simulator-based challenge would not transfer well to a physical robot. We find that this gap is largely due to AI agents learning to 'cheat' by exploiting simulator imperfections: specifically, the way Habitat allows for 'sliding' along walls on collision. Essentially, the virtual robot is capable of cutting corners, leading to unrealistic shortcuts through non-navigable spaces. Naturally, such exploits do not work in the real world where the robot stops on contact with walls. Our experiments show that it is possible to optimize simulation parameters to enable robots trained in imperfect simulators to generalize learned skills to reality (e.g. improving $SRCC_{Succ}$ from 0.18 to 0.844).

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “AI2-THOR, which are primarily designed for visual navigation, are also equipped with robotic embodiment for ease of deployment in their real-world counterparts such as iGibson’s Castro [37] and RoboTHOR [31] respectively.”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, background): “Note that in contrast to [3,5], we increased the angles associated with turn-left and turn-right actions from 10 ◦ to 30 ◦ degrees due to the concerns described in [4].”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, background): “Moreover still, findings and improvements in PointNav with GPS+Compass translate to improved performance on other tasks (PointNav without GPS+Compass [9, 19], ObjectGoal navigation [3, 6], RoomGoal navigation [17]) and to navigation by real robots [14].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “Future work should also assess if recent algorithmic progress in Habitat and RoboTHOR actually translates to real-world gains (in the spirit of prior work that indicates it does [34]).”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, background): “To this end, many works study the simulation to real transfer in the visual navigation domain [Bharadhwaj et al., 2019], while in [Kadian et al., 2019], the authors suggest to evaluate simulators in terms of how likely the performance improvement achieved on them can hold in reality, rather than…”
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, background): “In contrast, our approach does not rely on external information and recent work has shown promising sim2real transferability for navigation agents trained in continuous simulations [14].”
Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation (2021, methodology): “Further, despite advances in sensor technology, modern robots still often use cameras with noisy images, low frame rate, and a narrow field-of-view [32].”
An Exploration of Embodied Visual Exploration (2020, unknown): “Finally, recent work shows promising results transferring from simulated environments (including Matterport3D) to real-world exploration and navigation, supporting simulators as a meaningful testbed for our benchmark [15,34].”
Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation (2021, methodology): “This translates to 30° per step (as the policy is polled at 1 Hz), which has shown to work for visual navigation using the LoCoBot in [9].”