Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search

By Tianhao Zhang, Gregory Kahn, Sergey Levine, Pieter Abbeel. 2015

447 citations recorded by the source corpus.

Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “ocesses. The trialand-error process may lead to serious damage to real systems. Most of the reinforcement learning applications in robotics are trained in simulation environments (Tai and Liu, 2016a; Zhang et al., 2016a). However, model learned in simulation alone unfortunately cannot be directly tr”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “A slightly different approach is guided policy search (GPS) [9, 8], which in addition to classic control benchmarks, has achieved success in flight [24, 7], and robot manipulation [10, 14, 13].”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, background): “…function approximators such as deep neural networks, RL can provide impressive results on tasks ranging from playing games (Mnih et al., 2015; Silver et al., 2016), to ﬂying and driving (Lillicrap et al., 2015; Zhang et al., 2016), to controlling robotic arms (Levine et al., 2016; Gu et al., 2017).”
PLATO: Policy learning using adaptive trajectory optimization (2016, methodology): “The motivation behind PLATO is most similar to the MPC variant of guided policy search (MPC-GPS) [12].”
Socially aware motion planning with deep reinforcement learning (2017, background): “While several recent works have applied deep RL to motion planning [22], [23], they are mainly focused on single agent navigation in unknown static environments, and with an emphasis on computing control inputs directly from raw”
Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning (2016, background): “While applications of RL are typically limited to discrete, low-dimensional domains, recent advances in Deep RL [20]– [23] have demonstrated human-level performance in complex, high-dimensional spaces.”
Sparse identification of nonlinear dynamics for model predictive control in the low-data limit (2017, unknown): “Nonlinear models based on machine learning, such as neural networks, are increasingly used due to advances in computing power, and recently deep reinforcement learning has been combined with MPC [36, 56], yielding impressive results in the large-data limit.”
Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators (2019, methodology): “The state-of-the-art approaches using variations of this idea are concerned with using learned local models [20], compound multistep controllers [21], and deep representation of the control policies [22].”
Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control (2021, methodology): “Deep neural network models have been increasingly used for deep MPC [83, 84]; however, they often rely on access to massive data sets, have limited ability to generalize, do not readily incorporate known physical constraints, and are computationally expensive.”
Deep-Learned Collision Avoidance Policy for Distributed Multiagent Navigation (2016, background): “ich learns control policies mapping raw image observations to torques at the robots motors. Deep neural networks have also been combined with model predictive control to control robotic systems [22], [23]. Ondruska et al. [24], [25] combined recurrent neural networks with convolutional operations to”
Learning a deep neural net policy for end-to-end control of autonomous vehicles (2017, background): “Since explicit mapping of observations to control actions has not to be determined a priori in a design process anymore, deep learning turns out to be very efficient for certain high level controls objectives [6].”
A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning (2019, background): “ent Control Flight Systems (ICFS) is an area of study that was born out of the necessity to address some of the limitations of PID controllers; where RL has gained signiﬁcant traction in recent years [30, 31]. One particular scenario were our proposed problem would arise is in using RL to control au”
A survey of deep learning techniques for autonomous driving (2019, background): “rently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness (2017, methodology): “[33] use DRL to learn obstacle avoidance policies for a simulated quadrotor.”
A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles (2017, methodology): “Nevertheless, recent works have proposed novel methods to learn low-level control policies from imperfect sensor data in simulation [41, 63].”
Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not (2016, methodology): “Most of the reinforcement learning applications in robotics are trained in simulation environments (Tai and Liu, 2016a; Zhang et al., 2016a).”