Distral: Robust multitask reinforcement learning

By Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Manfred Otto Heess, Razvan Pascanu. 2017

628 citations recorded by the source corpus.

Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tasks. In practice, however, this is not usually observed, because gradients from different tasks can interfere negatively, making learning unstable and sometimes even less data efficient. Another issue is the different reward schemes between tasks, which can easily lead to one task dominating the learning of a shared model. We propose a new approach for joint training of multiple tasks, which we refer to as Distral (Distill & transfer learning). Instead of sharing parameters between the different workers, we propose to share a "distilled" policy that captures common behaviour across tasks. Each worker is trained to solve its own task while constrained to stay close to the shared policy, while the shared policy is trained by distillation to be the centroid of all task policies. Both aspects of the learning process are derived by optimizing a joint objective function. We show that our approach supports efficient transfer on complex 3D environments, outperforming several related methods. Moreover, the proposed learning process is more robust and more stable---attributes that are critical in deep reinforcement learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “In reinforcement learning, knowledge distillation has been used to regularize multi-task agents (Parisotto et al., 2016; Teh et al., 2017).”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “Advances in multi-task learning have been developed in the context of vision [5,20,32,42,52,53], language [10, 25, 26, 31, 37], and robotics [18, 34, 46].”
A Survey on Multi-Task Learning (2017, background): “In [141], each task has its own policy which is constrained to be close to a shared policy that is trained by distillation to be the centroid of all task policies.”
Knowledge Flow: Improve Upon Your Teachers (2019, background): “…particularly in reinforcement learning, teachers have also been considered in one way or another by Rusu et al. (2016b); Fernando et al. (2017); Wang et al. (2017); Li & Hoiem (2016); Bengio et al. (2009); Patel et al. (2015); Chen & Liu (2016); Teh et al. (2017); Parisotto et al. (2016).”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Policy distillation can be used then to learn several policies separately and simultaneously, and distill them into a single model as in the distral algorithm [Teh et al., 2017].”
Transfer Learning with Neural AutoML (2018, background): “Simultaneous multitask training can facilitate learning between tasks with a common structure, though retaining knowledge effectively across tasks is still an active area of research [29, 30].”
Recurrent Independent Mechanisms (2019, background): “There is some work done trying in Reinforcement learning, trying to learn default policies, which have shown to improve transfer and generalization in multi-task RL (Teh et al., 2017; Goyal et al., 2019a).”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “However, it is very difﬁcult to accumulate multiple skills using just one policy network Teh et al. (2017).”
Policy Consolidation for Continual Reinforcement Learning (2019, background): “Bidirectional distillation is also employed in (Teh et al., 2017), but for transfer learning in a multitask context.”
Progress & Compress: A scalable framework for continual learning (2018, methodology): “We also evaluate our method on 8 navigation tasks in 3D environments inspired by experiments with Distral (Teh et al., 2017).”
CoMPS: Continual Meta Policy Search (2021, positive): “There are a number of non-metalearning based methods that used behavioral cloning, distillation, and self-imitation for re-integrating previous experience (Rusu et al., 2015; Parisotto et al., 2016; Levine et al., 2016; Teh et al., 2017; Ghosh et al., 2018; Berseth et al., 2018).”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, background): “Meta-RL [9, 11, 15, 8] and multi-task RL [27, 36, 40, 43] also seek to accelerate learning by reusing information from different tasks, but in those settings the agent does not handle tasks arriving sequentially and the consequent problem of catastrophic forgetting.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks (Teh et al., 2017; Galashov et al., 2019; Tirumala et al., 2019; Jaques et al., 2017).”
Gradient Surgery for Multi-Task Learning (2020, methodology): “A different set of multi-task learning approaches aim to decompose the problem into multiple local problems, often corresponding to each task, that are significantly easier to learn, akin to divide and conquer algorithms [32, 50, 42, 56, 21, 13].”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “We highlight as potential candidates policy distillation (Parisotto, Ba, and Salakhutdinov 2015; Rusu et al. 2015; Schmitt et al. 2018; Teh et al. 2017) and active sampling of the task distribution the agent trains on (Sharma and Ravindran 2017).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Together, the combination of multitask and transfer learning can improve the sample efficiency and robustness of current DRL algorithms [140].”
Neural probabilistic motor primitives for humanoid control (2018, background): “This is distinct from activation matching or KL matching where the emphasis is on matching the action/activation distribution for a particular state (Rusu et al., 2015; Teh et al., 2017).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “This progressive growing of the set of agents on multi-task spaces is also related to progressive learning (Furlanello et al., 2018; Rusu et al., 2016; Schwarz et al., 2018), as well as multi-task learning with knowledge sharing (Teh et al., 2017).”