Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning

By Richard Liaw, Sanjay Krishnan, Animesh Garg, Daniel Crankshaw, Joseph E. Gonzalez, Ken Goldberg. 2017

26 citations recorded by the source corpus.

Rather than learning new control policies for each new task, it is possible, when tasks share some structure, to compose a "meta-policy" from previously learned policies. This paper reports results from experiments using Deep Reinforcement Learning on a continuous-state, discrete-action autonomous driving simulator. We explore how Deep Neural Networks can represent meta-policies that switch among a set of previously learned policies, specifically in settings where the dynamics of a new scenario are composed of a mixture of previously learned dynamics and where the state observation is possibly corrupted by sensing noise. We also report the results of experiments varying dynamics mixes, distractor policies, magnitudes/distributions of sensing noise, and obstacles. In a fully observed experiment, the meta-policy learning algorithm achieves 2.6x the reward achieved by the next best policy composition technique with 80% less exploration. In a partially observed experiment, the meta-policy learning algorithm converges after 50 iterations while a direct application of RL fails to converge even after 200 iterations.

Subfields: Artificial Intelligence, Automotive Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

Citations in the corpus

Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, background): “Authors in (Liaw et al., 2017) study the policy composition problem where composing previously learned basis policies, e.g., driving in different conditions or over different terrain types, the goal is to be able to reuse them for a novel task that is a mixture of previously seen dynamics by…”
Multi-Level Discovery of Deep Options (2017, background): “Early work in hierarchical control demonstrated the advantages of hierarchical structures by handcrafting hierarchical policies [19] and by learning them given various manual specifications: state abstractions [20, 21, 22, 23], a set of waypoints [24], low-level skills [25, 26, 27], a set of finite-”