Distilling Policy Distillation

By Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, Max Jaderberg. 2019

179 citations recorded by the source corpus.

The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the widespread use and conceptual simplicity of distillation, many different formulations are used in practice, and the subtle variations between them can often drastically change the performance and the resulting objective that is being optimised. In this work, we rigorously explore the entire landscape of policy distillation, comparing the motivations and strengths of each variant through theoretical and empirical analysis. Our results point to three distillation techniques, that are preferred depending on specifics of the task. Specifically a newly proposed expected entropy regularised distillation allows for quicker learning in a wide range of situations, while still guaranteeing convergence.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, methodology): “Therefore, we successfully implemented behavioral cloning in the spirit of the DAGGER [88] algorithm (sometimes also called policy distillation [23]) to efficiently initialize new policies with a level of performance very close to the teacher policy.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “Eventually, the local models are combined into a single, multi-task policy using different distillation techniques (outlined in (Hinton et al., 2015; Czarnecki et al., 2019)).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “…that higher performance and faster training can be achieved in deep RL by first training an agent, then subsequently training a new agent on the identical task whilst performing policy distillation from the first agent (Czarnecki et al., 2019; Furlanello et al., 2018; Schmitt et al., 2018).”
Dual Policy Distillation (2020, unknown): “Recently, works have been proposed to study how we can transfer knowledge from one or more teacher models to a student model so that we can train an agent based on a pre-trained expert model [Rusu et al. , 2016; Czarnecki et al. , 2019].”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, background): “More recently, [14, 61] further extend this idea to enable better sample efﬁciency and [51, 70] apply it to multi-task DRL.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “For the ﬁrst four billion steps of training, we use an additional distillation loss (Schmidhuber, 1992; Schmitt et al., 2018; Czarnecki et al., 2019) to guide AdA’s learning with the policy of a pre-trained teacher, in a process known as kickstarting; iterating this process leads to a generational…”
Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control (2020, result): “SmallTorso and HalfCheetahBigTorso, respectively. On average (over all the tasks), KTM-DRL beats the ideal solution by 2%, which validates its effectiveness. It is worth to mention that related works [3, 14, 15] also made similar observations that a single multi-task agent may surpass their task-spe”
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation (2020, unknown): “It is often used for learning from multiple source tasks, where a student agent is trained to mimic the behaviour of one or multiple teacher policies [6].”
Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings (2020, background): “Second, recent studies on policy distillation [9], [10] have empirically shown that, with effective knowledge representation, SL can generalize to sequential prediction tasks.”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, methodology): “Policy distillation has been applied to RL (Czarnecki et al., 2019), for example for multi-task learning and compression (Teh et al., 2017; Rusu et al., 2015; Parisotto et al., 2016), imitation learning (Ross et al., 2011), or faster training (Schmitt et al., 2018; Ghosh et al., 2018).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “…π E i ( a | s ) and the student π θ , approximated using the dataset (cid:5) Another policy distillation approach is student distillation [51], [60], which is resemblant to teacher distillation except that during the optimization step, the objec-tive expectation is taken over trajectories…”