Policies Modulating Trajectory Generators

By Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, Vincent Vanhoucke. 2018

151 citations recorded by the source corpus.

We propose an architecture for learning complex controllable behaviors by having simple Policies Modulate Trajectory Generators (PMTG), a powerful combination that can provide both memory and prior knowledge to the controller. The result is a flexible architecture that is applicable to a class of problems with periodic motion for which one has an insight into the class of trajectories that might lead to a desired behavior. We illustrate the basics of our architecture using a synthetic control problem, then go on to learn speed-controlled locomotion for a quadrupedal robot by using Deep Reinforcement Learning and Evolutionary Strategies. We demonstrate that a simple linear policy, when paired with a parametric Trajectory Generator for quadrupedal gaits, can induce walking behaviors with controllable speed from 4-dimensional IMU observations alone, and can be learned in under 1000 rollouts. We also transfer these policies to a real robot and show locomotion with controllable forward velocity.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

Learning quadrupedal locomotion over challenging terrain (2020, positive): “We use the Policies Modulating Trajectory Generators (PMTG) architecture (34) to provide priors on motion generation.”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “Other than domain randomization, other parallel research directions for sim-to-real transfer exist and have been demonstrated to be promising research areas as well ([21], [22], [23], [24], [11]).”
RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “The most comparable work in terms of robust performance of RL policies for legged locomotion in the real-world is that of Lee et al. [32] which, unlike our work, relies on hand-coded domain knowledge of predeﬁned trajectory generator [25] and motor models [23].”
Learning to Walk via Deep Reinforcement Learning (2018, background): “, [24]), end-to-end methods that make minimal prior assumptions are broadly applicable, and developing such techniques will make it more scalable to acquire gaits for diverse robots in diverse conditions.”
Data Efficient Reinforcement Learning for Legged Robots (2019, methodology): “Similar to [13], TGs output periodic trajectories in the extension space of each leg (Fig.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “PMTG subdivides the controller into an open-loop and a feedback component.”