(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image

By Fereshteh Sadeghi, Sergey Levine. 2016

898 citations recorded by the source corpus.

Deep reinforcement learning has emerged as a promising and powerful technique for automatically acquiring control policies that can process raw sensory inputs, such as images, and perform complex behaviors. However, extending deep RL to real-world robotic tasks has proven challenging, particularly in safety-critical domains such as autonomous flight, where a trial-and-error learning process is often impractical. In this paper, we explore the following question: can we train vision-based navigation policies entirely in simulation, and then transfer them into the real world to achieve real-world flight without a single real training image? We propose a learning method that we call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models. Our method uses single RGB images from a monocular camera, without needing to explicitly reconstruct the 3D geometry of the environment or perform explicit motion planning. Our learned collision avoidance policy is represented by a deep convolutional neural network that directly processes raw monocular images and outputs velocity commands. This policy is trained entirely on simulated images, with a Monte Carlo policy evaluation algorithm that directly optimizes the network's ability to produce collision-free flight. By highly randomizing the rendering settings for our simulated training set, we show that we can train a policy that generalizes to the real world, without requiring the simulator to be particularly realistic or high-fidelity. We evaluate our method by flying a real quadrotor through indoor environments, and further evaluate the design choices in our simulator through a series of ablation studies on depth prediction. For supplementary video see: this https URL

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, methodology): “Sadeghi and Levine’s work [37] is the most similar to our own.”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision [64, 34, 28, 4, 15, 24, 33, 59, 14] and control [60, 45, 46, 20, 54].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…and autonomous driving, a rapidly growing research topic is the study of simulation to real-world transfer: training policies with reinforcement learning in simulation and then transferring these policies into the real world (Sadeghi and Levine, 2017; Tan et al., 2018; Chebotar et al., 2019).”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “Domain randomization was used to transfer object pose estimators [106] and vision policies for fly drones [93].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “For ﬁdelity , simulators that are world-based scene will undoubtedly outperform game-based scene simulator in simulation to real tasks [27], [112].”
Deep Drone Racing: Learning Agile Flight in Dynamic Environments (2018, background): “While being independent of any global map and position estimate, these methods are not directly applicable to our specific problem due to their high computational complexity [27, 11], their low maximum speed [13] or the inherent difficulties of generalizing to 3D motions [25, 4, 24].”
A 64-mW DNN-Based Visual Navigation Engine for Autonomous Nano-Drones (2018, background): “Recent results have shown that much lighter algorithms, based on convolutional neural networks (CNNs), are sufﬁcient for enabling basic reactive navigation of small drones, even without a map of the environment [2], [17], [18], [19], [20].”
The limits and potentials of deep learning for robotics (2018, background): “Another promising recent direction is domain randomization [97], [118].”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, methodology): “While these methods have been used to learn robot navigation policies, they often require simulation experience [19], [20].”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, background): “However, these methods have typically focused on pure geometric reasoning, require on-policy data, and often utilize simulation due to constraints such as sample efficiency [28, 8, 29, 30, 31, 32].”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have tried to address this by developing simulations from which large amounts of diverse data can be collected [1, 2], or by attempting to make robot learning algorithms more data efﬁcient [3, 4].”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “Domain randomization of textures was used in [7] to learn to fly a real quadcopter by training an image based policy entirely in simulation.”
Visual Imitation Made Easy (2020, background): “It has also been extensively used in context of RL, where domain randomization is proposed to transfer learning from simulation to real world [36, 37, 38].”
Reinforcement Learning with Augmented Data (2020, methodology): “While not directly known for data augmentation in reinforcement learning, the following ideas can be viewed as techniques to diversify the data used to train an RL agent: Domain Randomization [29, 30] is a simple data augmentation technique primarily used for transferring policies from simulation to”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, background): “Several works [32, 33] show promise in this direction.”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “While the main goal of this work is to study how modelfree RL can be used to learn complex policies directly on real hardware, we also evaluated training in simulation and transfer, employing randomization to allow for transfer [22], [23].”