Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

By Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov. 2015

654 citations recorded by the source corpus.

The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, neutral): “In fact, pretraining is in some cases worse than random initialization, a fact observed in prior RL work (Parisotto et al., 2016).”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “There has been a growing interest in using deep neural networks for multi-task learning and transfer learning (Parisotto et al., 2015; Rusu et al., 2015; 2016a; Devin et al., 2016; Rusu et al., 2016b).”
An Overview of Multi-task Learning (2018, methodology): “In [73], an actor–mimic method, which is a combination of deep reinforcement learning andmodel compression techniques, is proposed to learn policy networks for multiple tasks.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “In reinforcement learning, knowledge distillation has been used to regularize multi-task agents (Parisotto et al., 2016; Teh et al., 2017).”
Overcoming catastrophic forgetting in neural networks (2016, methodology): “…learning approaches to continual learning have either relied on either adding capacity to the network [Ring, 1998, Rusu et al., 2016] or on learning each task in separate networks, which are then used to train a single network that can play all games[Rusu et al., 2015, Parisotto et al., 2015].”
What can the brain teach us about building artificial intelligence? (2016, background): “This has been the rationale behind multi-task learning or transfer learning, a strategy with a long history that has shown some promising results recently with deep networks (e.g., Donahue et al., 2013; Luong, Le, Sutskever, Vinyals, & Kaiser, 2015; Parisotto et al., 2016).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “There are attempts to apply transfer learning in the ALE (Rusu et al., 2016; Parisotto et al., 2016).”
Progressive Neural Networks (2016, methodology): “Performance is evaluated by measuring the area under the learning curve (average score per episode during training), rather than final score.”
A Survey on Multi-Task Learning (2017, methodology): “In [137], the proposed ActorMimic method combines both deep reinforcement learning and”
Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “…transfer have been considered using a variety of techniques, for instance, ﬁne-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Knowledge transfer techniques like distillation (Bengio, 2012; Rusu et al., 2016; Parisotto et al., 2016; Teh et al., 2017) have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task…”
Graying the black box: Understanding DQNs (2016, background): “The ability to understand the hierarchical structure of the policy can help in distilling it into a simpler architec- ture (Rusu et al., 2015; Parisotto et al., 2015).”
Deep Generative Dual Memory Network for Continual Learning (2017, background): “Such distillation can sometimes improve performance on individual tasks if they share structure and at other times mitigate inter-task interference due to refinement of learnt functions while distilling (Parisotto et al., 2016).”
Progress & Compress: A scalable framework for continual learning (2018, methodology): “Examples of this technique include transfer from Deep Q-Networks (Parisotto et al., 2015) or curriculum learning in memory models (Graves et al.”
CoMPS: Continual Meta Policy Search (2021, background): “Both terms describe the process of learning tasks in sequence while avoiding the problem of forgetting (French, 1999; Rusu et al., 2015; Parisotto et al., 2016; Rusu et al., 2016; Chen et al., 2015; Tessler et al., 2016; Aljundi et al., 2019).”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, background): “Meta-RL [9, 11, 15, 8] and multi-task RL [27, 36, 40, 43] also seek to accelerate learning by reusing information from different tasks, but in those settings the agent does not handle tasks arriving sequentially and the consequent problem of catastrophic forgetting.”