AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
By Shital Shah, Debadeepta Dey, Chris Lovett, Ashish Kapoor. 2017
2,474 citations recorded by the source corpus.
Developing and testing algorithms for autonomous vehicles in real world is an expensive and time consuming process. Also, in order to utilize recent advances in machine intelligence and deep learning we need to collect a large amount of annotated training data in a variety of conditions and environments. We present a new simulator built on Unreal Engine that offers physically and visually realistic simulations for both of these goals. Our simulator includes a physics engine that can operate at a high frequency for real-time hardware-in-the-loop (HITL) simulations with support for popular protocols (e.g. MavLink). The simulator is designed from the ground up to be extensible to accommodate new types of vehicles, hardware platforms and software protocols. In addition, the modular design enables various components to be easily usable independently in other projects. We demonstrate the simulator by first implementing a quadrotor as an autonomous vehicle and then experimentally comparing the software components with real-world flights.
Subfields: Automotive Engineering, Control and Systems Engineering, Management Science and Operations Research, ML Systems & Hardware, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Automotive Engineering
- Sergey I. Nikolenko
- Davide Scaramuzza
- Vladlen Koltun
- Dengxin Dai
- Luc Van Gool
- Matthias Müller
- Longlong Jing
- Yingli Tian
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhinav Kumar Gupta
- Lerrel Pinto
- Vladlen Koltun
- Ajay Mandlekar
- Ruslan Salakhutdinov
In Management Science and Operations Research
- Pieter Abbeel
- Ruslan Salakhutdinov
- Sergey I. Nikolenko
- Vladlen Koltun
- Sergey Levine
- Josh Tenenbaum
- Abhinav Kumar Gupta
- Jiarong Lin
In ML Systems & Hardware
- Sergey I. Nikolenko
- Josh Tenenbaum
- Yingli Tian
- Jiarong Lin
- Longlong Jing
- Fu Zhang
- Chen Wang
- Yuheng Qiu
In Robotics & Control
- Sergey Levine
- Vladlen Koltun
- Sergey I. Nikolenko
- Abhinav Kumar Gupta
- Roozbeh Mottaghi
- Pieter Abbeel
- Lerrel Pinto
- Alexey Dosovitskiy
Citations in the corpus
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “In response to this challenge, the computer vision community has developed several photorealistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [5, 6, 8, 9, 13, 14, 17, 20, 22, 29, 31, 34, 35, 37, 4”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Simulator Description CARLA [79] Urban simulator, Camera & LIDAR streams, with depth & semantic segmentation, Location information TORCS [103] Racing Simulator, Camera stream, agent positions, testing control policies for vehicles AIRSIM [104] Camera stream with depth and semantic segmentation, supp”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Modalities Framerate Target Use Case Gazebo (Koenig et al., 2004) Sensor Plugins 10s+FPS General Purposes Vrep (Rohmer et al., 2013) Sensor Plugins 10s+FPS General Purposes Airsim (Shah et al., 2017) Depth/Color/Semantics 20s+FPS Autonomous Driving Carla (Dosovitskiy et al., 2017) Depth/Color/Semant”
- Lifelong Graph Learning (2020, positive): “The sequences are rendered in AirSim [38], a photo-realistic simulator, which features modeled environments with various themes including urban, rural, nature, domestic, public, sci-fi, etc .”
- Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, methodology): “In driving, several simulators like CARLA [27] and AirSim [28] promise easier self driving research.”
- MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “UAV control has been studied using AirSim (Shah et al., 2017) and UE4Sim (M¨uller et al., 2017).”
- R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package (2021, methodology): “14, we built the car and drone simulator with the AirSim [32], and in Fig.”
- IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks (2019, background): “Recently, many simulated environments have been introduced in diverse applications, such as autonomous driving [12, 13], indoor navigation [14, 15], continuous control [16–18], and recommendation systems [19].”
- Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, methodology): “…Driving Simulator with Camera, LiDAR, Depth & Semantic segmentation Racing Simulator TORCS (Wymann et al., 2000) Testing control policies for vehicles AIRSIM (Shah et al., 2018) Resembling CARLA with support for Drones GAZEBO (Koenig and Howard, 2004) Multi-robo simulator for planning & control”
- Integrating Physics-Based Modeling with Machine Learning: A Survey (2020, background): “Then, in autonomous vehicle training, Shah et al. [230] showed that pre-training the driving algorithm in a simulator built on a video game physics engine can drastically lessen data needs.”
- Mutual Alignment Transfer Learning (2017, background): “Recent developments aim at designing visually more similar environments [9, 10] and current research targets adapting policies to be invariant with respect to differences between the observation spaces of simulator and real platform [11, 12, 13, 14].”
- Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020, background): “In recent years, simulators have been progressing towards more realistic scenarios and physics engines: Air-sim [13], CARLA [14], RotorS [15], [16], and others [17].”
- Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications (2017, methodology): “build on top of our work, use plugins such as AirSim (Shah et al. 2017) or UnrealCV (Qiu et al.”
- Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning (2019, background): “High-fidelity Autonomous driving simulators like CARLA [4] and AirSim [23] provide a simulation platform for training Deep RL agents in singe-agent driving scenarios.”
- UnrealCV: Virtual Worlds for Computer Vision (2017, methodology): “Unreal Engine has been used in a few research projects [5, 9].”
- Cross-View Policy Learning for Street Navigation (2019, background): “There has been a growing interest in building and benchmarking visual navigation using complex simulators [21, 38] or photorealistic indoor environments [31].”
- Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “In addition, the AirSim [384] open-source”