PLATO: Policy learning using adaptive trajectory optimization

By Gregory Kahn, Tianhao Zhang, Sergey Levine, Pieter Abbeel. 2016

140 citations recorded by the source corpus.

Policy search can in principle acquire complex strategies for control of robots and other autonomous systems. When the policy is trained to process raw sensory inputs, such as images and depth maps, it can also acquire a strategy that combines perception and control. However, effectively processing such complex inputs requires an expressive policy class, such as a large neural network. These high-dimensional policies are difficult to train, especially when learning to control safety-critical systems. We propose PLATO, a continuous, reset-free reinforcement learning algorithm that trains complex control policies with supervised learning, using model-predictive control (MPC) to generate the supervision, hence never in need of running a partially trained and potentially unsafe policy. PLATO uses an adaptive training method to modify the behavior of MPC to gradually match the learned policy in order to generate training samples at states that are likely to be visited by the learned policy. PLATO also maintains the MPC cost as an objective to avoid highly undesirable actions that would result from strictly following the learned policy before it has been fully trained. We prove that this type of adaptive MPC expert produces supervision that leads to good long-horizon performance of the resulting policy. We also empirically demonstrate that MPC can still avoid dangerous on-policy actions in unexpected situations during training. Our empirical results on a set of challenging simulated aerial vehicle tasks demonstrate that, compared to prior methods, PLATO learns faster, experiences substantially fewer catastrophic failures (crashes) during training, and often converges to a better policy.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “We can use the following lemma from Kahn et al. (2016), which is adapted from Ross et al. (2011) and Schulman et al. (2015).”
Deep Drone Racing: Learning Agile Flight in Dynamic Environments (2018, background): “Known output representations range from predicting discrete navigation commands [15, 17]— which enables high robustness but leads to low agility—to direct control [14]—which can lead to highly agile control, but suffers from high sample complexity.”
Uncertainty-Aware Reinforcement Learning for Collision Avoidance (2017, background): “Prior work has considered safety at training time by using model-predictive control (MPC) with ground truth state information [11].”
Agile Autonomous Driving using End-to-End Deep Imitation Learning (2017, background): “box oracle that can provide the desired actions when queried, as opposed to the case considered in [9] where the expert can be modified to accommodate the learning progress.”
P3O: Policy-on Policy-off Policy Optimization (2019, background): “This is elaborated upon by Lemma 4 which follows from the time-dependent state distribution bound proved in (Schulman et al., 2015a; Kahn et al., 2017).”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “Some of these works (Kahn et al. 2017; Giusti et al. 2016), focus on the problem of learning controllers for effectively maneuvering around obstacles directly from raw sensor data.”
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction (2017, background): “…this oracle may come from a human expert guiding the robot during the training phase (Abbeel & Ng, 2004) or from an optimal MDP solver (Ross et al., 2011; Kahn et al., 2016) that either may be too slow to use at test time or leverages information unavailable at test time (e.g., ground truth).”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “A slightly different approach is guided policy search (GPS) [9, 8], which in addition to classic control benchmarks, has achieved success in flight [24, 7], and robot manipulation [10, 14, 13].”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “RL approaches [68,1,20,33,44,92,62,86] have been proposed to address navigation in synthetic layouts like mazes, arcade games and other visual environments [100,8,47,54,43,84].”
Kernelized movement primitives (2017, unknown): “As an effective way to measure the distance between two probabilistic distributions, KL-divergence was exploited in policy search (Peters et al. 2010; Kahn et al. 2017), trajectory optimization (Levine and Abbeel 2014) and imitation learning (Englert et al. 2013).”
DroNet: Learning to Fly by Driving (2018, unknown): “jectories [10], or model predictive control [12].”
Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, background): “similar to [4], however, that is not a strict requirement and human input is also easy to integrate.”
Two Body Problem: Collaborative Visual Task Completion (2019, background): “Consequently the task is well suited for an RL formulation, a perspective which has become popular recently [62, 1, 16, 17, 33, 42, 86, 59, 5, 8, 90, 25, 36, 91, 37].”
Agile Off-Road Autonomous Driving Using End-to-End Deep Imitation Learning (2017, background): “Note that here we assume the expert is given as a black box oracle that can provide the desired actions when queried, as opposed to the case considered in [12] where the expert can be modiﬁed to accommodate to the learning progress.”
Guided Meta-Policy Search (2019, unknown): “Our proof follows a similar approach to prior work that analyzes the convergence of imitation algorithms with aggregation (Ross et al., 2011; Kahn et al., 2016), but extends these results into the meta-learning setting.”
Output Range Analysis for Deep Feedforward Neural Networks (2018, background): “Neural networks are increasingly used as models of physical dynamics and feedback control laws to achieve objectives such as safety, reachability and stability [13].”