Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control

By Glen Berseth, Kevin Xie, Paul Cernek, Michiel van de Panne. 2018

60 citations recorded by the source corpus.

Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a specific skill and its associated state distribution. We extend policy distillation methods to the continuous action setting and leverage this technique to combine expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain. We also introduce an input injection method for augmenting an existing policy network to exploit new input features. Lastly, our method uses transfer learning to assist in the efficient acquisition of new skills. The combination of these methods allows a policy to be incrementally augmented with new skills. We compare our progressive learning and integration via distillation (PLAID) method against three alternative baselines.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

Progress & Compress: A scalable framework for continual learning (2018, methodology): “PLAiD (Berseth et al., 2018) is method with similarities with our approach.”
CoMPS: Continual Meta Policy Search (2021, positive): “There are a number of non-metalearning based methods that used behavioral cloning, distillation, and self-imitation for re-integrating previous experience (Rusu et al., 2015; Parisotto et al., 2016; Levine et al., 2016; Teh et al., 2017; Ghosh et al., 2018; Berseth et al., 2018).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2015) from past tasks when learning a new task to prevent catastrophic forgetting as in (Rusu et al., 2015; Li and Hoiem, 2016; Riemer et al., 2016; Espeholt et al., 2018; Schwarz et al., 2018; Berseth et al., 2018; Kaplanis et al., 2019; Traoré et al., 2019; Tirumala et al., 2019; Igl et al., 202”
One Big Net For Everything (2018, background): “Compare a recent study of incremental skill learning with feedforward networks [5].”
Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting (2018, background): “For example, PLAID [28] uses distillation to merge two expert models.”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, background): “Recently, progressive reinforcement learning [3] has been proposed to distill skills from multiple expert policies.”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, result): “Similar for the “One Big Net” [54] and a recent study of incremental skill learning with feedforward networks [4].”
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation (2021, methodology): “Within RL, distillation has most often been used as a method for stabilizing multitask learning (Rusu et al., 2015; Parisotto et al., 2015; Teh et al., 2017; Berseth et al., 2018).”
Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real (2019, methodology): “It is used to train policies to play multiple Atari games [12, 13] and simulated humanoids to traverse different terrains [14].”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Deep RL also has been used to learn locomotion control policies [29, 52, 43, 9, 55], mostly in simulated environments.”
Kickstarting Deep Reinforcement Learning (2018, background): “Typically one gathers experience from expert policies, which are then used to train a student model using supervised learning (Rusu et al., 2015; Berseth et al., 2018).”
Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning (2019, background): “It often shows the degradation of the performance in individual tasks in case of learning multiple skills at once (e.g., Multitasker in [2]).”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, methodology): “When transferring to tasks with additional goal inputs g that are not present during training, the networks are augmented with additional inputs using the input injection method from Berseth et al. [3], which adds additional inputs to the network without modifying the initial behavior of the model.”