Kickstarting Deep Reinforcement Learning

By Simon Schmitt, Jonathan J. Hudson, Augustin Žídek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan and 1 more. 2018

151 citations recorded by the source corpus.

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Red Teaming Language Models with Language Models (2022, methodology): “To prevent RL from collapsing to a single, highreward generation, we add a loss term to penalize KL divergence between pr(x) and initialization’s distribution over next tokens (Jaques et al., 2017; Schmitt et al., 2018; Jaques et al., 2019; Ziegler et al., 2019).”
Teaching language models to support answers with verified quotes (2022, methodology): “To prevent RL from collapsing to a single, high-reward generation, we add a loss term to penalize KL divergence between 𝑝𝑟 (𝑥) and initialization’s distribution over next tokens (Jaques et al., 2017, 2019; Schmitt et al., 2018; Ziegler et al., 2019).”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, background): “For example, the training from scratch or hyperparameter tuning of such networks can be prohibitively expensive or time consuming (Schmitt et al. 2018).”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “We highlight as potential candidates policy distillation (Parisotto, Ba, and Salakhutdinov 2015; Rusu et al. 2015; Schmitt et al. 2018; Teh et al. 2017) and active sampling of the task distribution the agent trains on (Sharma and Ravindran 2017).”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Since then, knowledge distillation has been widely adopted in a variety of learning tasks (Yim et al. 2017; Yu et al. 2017; Schmitt et al. 2018; Chen et al. 2017).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “…that higher performance and faster training can be achieved in deep RL by first training an agent, then subsequently training a new agent on the identical task whilst performing policy distillation from the first agent (Czarnecki et al., 2019; Furlanello et al., 2018; Schmitt et al., 2018).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, neutral): “The pixel-control IMPALA agents shown here were carefully tuned for DMLab and are similar to the “experts” used in Schmitt et al. (2018); in all cases these results match or exceed previously published results for IMPALA (Espeholt et al., 2018; Kapturowski et al., 2019).”
Self-Attentional Credit Assignment for Transfer in Reinforcement Learning (2019, unknown): “Previous works aimed at making the training of an agent in the same task more sample-efficient by using a pretrained model as a teacher (Rusu et al., 2016a; Schmitt et al., 2018).”
Information asymmetry in KL-regularized RL (2019, methodology): “The speed-up from regularizing to the pretrained expert is signiﬁcant, however it corresponds to regularizing against an existing solution and can thus primarily be used as a method to speed-up the experiment cycles, as it was demonstrated in the kickstarting framework (Schmitt et al., 2018).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “For the ﬁrst four billion steps of training, we use an additional distillation loss (Schmidhuber, 1992; Schmitt et al., 2018; Czarnecki et al., 2019) to guide AdA’s learning with the policy of a pre-trained teacher, in a process known as kickstarting; iterating this process leads to a generational…”
Attentive Multi-Task Deep Reinforcement Learning (2019, background): “[27] also build on the idea of policy distillation”
A Survey of Deep Reinforcement Learning in Video Games (2019, methodology): “Schmitt et al. [88] present a method which uses teacher agents to kickstart the training of a new student agent.”
Adapting Auxiliary Losses Using Gradient Similarity (2018, background): “, 2015), to guide the main task in its initial learning phase (Schmitt et al., 2018), although it might be difficult to find a suitable strategy that combines the main and auxiliary losses and/or smoothly transition between them.”
Backplay: "Man muss immer umkehren" (2018, unknown): “All of our attempts to use reward signal to improve the agent’s performance (over that of the behaviorally cloned agent) were unsuccessful, even after incorporating tricks in the literature such as those found in Schmitt et al. (2018).”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, methodology): “Policy distillation has been applied to RL (Czarnecki et al., 2019), for example for multi-task learning and compression (Teh et al., 2017; Rusu et al., 2015; Parisotto et al., 2016), imitation learning (Ross et al., 2011), or faster training (Schmitt et al., 2018; Ghosh et al., 2018).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “A similar approach was proposed by [107] which”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, background): “More recently, [14, 61] further extend this idea to enable better sample efﬁciency and [51, 70] apply it to multi-task DRL.”
Deep neuroethology of a virtual rodent (2019, methodology): “Consequently, we present results arising from training a singletask expert on the escape task and training the multi-task policies using kickstarting for that task (Schmitt et al., 2018), with a weak coefficient (.”
Distilling Policy Distillation (2019, methodology): “If we choose `(τ |θ) = ∑|τ | t=1 H (π(τt)‖πθ(τt)) we recover the onpolicy distillation updates used in techniques such as kickstarting [26] and Mix&Match [5].”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Schmitt et al. (2018) use a pretrained policy as the default policy; other works jointly learn the policy and default policy to capture reusable behaviour from experience (Teh et al., 2017; Czarnecki et al., 2018; Galashov et al., 2019; Grau-Moya et al., 2019).”