Mix&Match - Agent Curricula for Reinforcement Learning

By Wojciech M. Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess, Simon Osindero, Razvan Pascanu. 2018

83 citations recorded by the source corpus.

We introduce MixM using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “Past methods have grown neural networks by incrementally training and freezing parts of the network [49], [50], [51].”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “Closer to ITER , Czarnecki et al. (2018) use policy distillation to learn a sequence of agents.”
Attentive Multi-Task Deep Reinforcement Learning (2019, methodology): “Czarnecki et al. [5] use multiple networks similar to our approach.”
Information asymmetry in KL-regularized RL (2019, unknown): “Czarnecki et al. (2018) also relies on a KL-regularized objective to ensure policies explored in a curriculum stay close to each other.”
Distilling Policy Distillation (2019, methodology): “…under the teacher policy [32, 27], and the other is to frame the learning as a per-timestep supervised learning problem, deﬁning the loss at each timestep to be the cross entropy between the teacher’s and student’s distributions over actions [2, 26, 5] There are two aspects worth discussing.”
A Survey of Deep Reinforcement Learning in Video Games (2019, background): “Mix & Match is a training framework that is designed to encourage effective and rapid learning in DRL agents [116].”
Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research (2019, background): “Researcher-controlled non-stationary dynamics in machine learning, known as curricula, can facilitate the acquisition of complex behaviors that would not be learnable otherwise, e.g. Asmuth et al. (2008); Bengio et al. (2009); Czarnecki et al. (2018); Narvekar (2017); Zaremba and Sutskever (2015).”
Growing Action Spaces (2019, methodology): “Empirically, in our settings, we find our approach to perform better and more consistently than an actor-critic algorithm modeled after Czarnecki et al. (2018), although we do not take on the significant additional computational requirements of population based training in any of our experiments.”
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation (2021, methodology): “The method most closely related to ALD is Mix&Match (Czarnecki et al., 2018), where an ensemble of policies is deﬁned such that each successive model in the ensemble has increasing capacity.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Schmitt et al. (2018) use a pretrained policy as the default policy; other works jointly learn the policy and default policy to capture reusable behaviour from experience (Teh et al., 2017; Czarnecki et al., 2018; Galashov et al., 2019; Grau-Moya et al., 2019).”