Policy Distillation

By Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, Raia Hadsell. 2015

888 citations recorded by the source corpus.

Abstract: Policies for complex visual tasks have been successfully learned with deep reinforcement learning, using an approach called deep Q-networks (DQN), but relatively large (task-specific) networks and extensive training are needed to achieve good performance. In this work, we present a novel method called policy distillation that can be used to extract the policy of a reinforcement learning agent and train a new network that performs at the expert level while being dramatically smaller and more efficient. Furthermore, the same method can be used to consolidate multiple task-specific policies into a single policy. We demonstrate these claims using the Atari domain and show that the multi-task distilled agent outperforms the single-task teachers as well as a jointly-trained DQN agent.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “There has been a growing interest in using deep neural networks for multi-task learning and transfer learning (Parisotto et al., 2015; Rusu et al., 2015; 2016a; Devin et al., 2016; Rusu et al., 2016b).”
Overcoming catastrophic forgetting in neural networks (2016, methodology): “…learning approaches to continual learning have either relied on either adding capacity to the network [Ring, 1998, Rusu et al., 2016] or on learning each task in separate networks, which are then used to train a single network that can play all games[Rusu et al., 2015, Parisotto et al., 2015].”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, background): “Properly handling such diversity might benefit from explicit exploration or even separate policies trained on each robot, with subsequent pooling based on policy distillation [42].”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “There are attempts to apply transfer learning in the ALE (Parisotto, Ba, & Salakhutdinov, 2016; Rusu et al., 2016).”
Progressive Neural Networks (2016, methodology): “Recently, there have been new methods proposed for multi-task or transfer learning with deep RL: [21, 17, 14].”
Born Again Neural Networks (2018, background): “In both cases, the idea is typically to transfer the knowledge of a high-capacity teacher with formidable performance to a more compact student [8, 9, 10].”
A Survey on Multi-Task Learning (2017, methodology): “In [152], a policy distillation method is proposed to not only train an efficient network to learn the policy of an agent but also consolidate multiple task-specific policies into a single policy.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Many studies proposed ideas to transfer knowledge of a teacher to a compact student (Ba and Caruana 2014; Rusu et al. 2015; Bucila, Caruana, and Niculescu-Mizil 2006; Urban et al. 2016).”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “, 2017), and finally the field of policy distillation in reinforcement learning (Rusu et al., 2016), where one requires observations from the original training environment which may not exist anymore.”
Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “Following their setup, we conduct experiments on MNIST, MNIST with digit ‘3’ missing in the training set, CIFAR-100, and ImageNet.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “In order to ﬁt reinforcement learning into a continual setting, we use a method called policy distillation [Rusu et al., 2015] that allows to transfer several policies learned sequentially into a single model.”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, methodology): “Using a technique termed policy distillation, (Rusu et al. 2015) shows the policy of an RL agent can be extracted to train a smaller, more efficient network to perform at expert level.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Knowledge transfer techniques like distillation (Bengio, 2012; Rusu et al., 2016; Parisotto et al., 2016; Teh et al., 2017) have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task…”
Deep Generative Dual Memory Network for Continual Learning (2017, background): “Connections to knowledge distillation: Previous works on (joint) multitask learning have also proposed approaches to learn individual tasks with small networks and then “distilling” them jointly into a larger network (Rusu et al., 2015).”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, methodology): “There are two types of Deep Skill Modules: (1) a DSN array (Figure 3, Module A) and (2) a multi-skill distillation network (Figure 3, Module B), our novel variation of policy distillation (Rusu et al. 2015) applied to learning skills.”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “…for the PC model, we adapt the model of (Kaplanis et al., 2018) by incorporating concepts from knowledge distillation (Hinton et al., 2015; Rusu et al., 2015) and from one of the proximal policy optimisation (PPO) algorithms for RL (Schulman et al., 2017) in order to implement…”
Progress & Compress: A scalable framework for continual learning (2018, background): “The consolidation is done via a distillation process (Hinton et al., 2015; Rusu et al., 2015), which is an effective mechanism for transferring knowledge from the active column to the knowledge base.”
CoMPS: Continual Meta Policy Search (2021, background): “Both terms describe the process of learning tasks in sequence while avoiding the problem of forgetting (French, 1999; Rusu et al., 2015; Parisotto et al., 2016; Rusu et al., 2016; Chen et al., 2015; Tessler et al., 2016; Aljundi et al., 2019).”