Sharing Knowledge in Multi-Task Deep Reinforcement Learning

By Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters. 2020

156 citations recorded by the source corpus.

We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

Subfields: Artificial Intelligence, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Joelle Pineau
2. Qiang Yang
3. Sergey Levine
4. Mohit Bansal
5. Yoshua Bengio
6. Yu Zhang
7. Jianfeng Gao
8. Luke S. Zettlemoyer

In Cognitive Neuroscience

1. Qiang Yang
2. Yu Zhang
3. Jieping Ye
4. Doina Precup
5. Joelle Pineau
6. Michael Crawshaw
7. Zhengping Che
8. Amy Zhang

Citations in the corpus

• A Survey on Multi-Task Learning (2017, background): “For example, in [161], sharing representations among tasks is analyzed with theoreti-”
• Invariant Causal Prediction for Block MDPs (2020, background): “D’Eramo et al. (2020) show the benefits of learning a shared representation in multi-task settings with an approximate value iteration bound and Brunskill and Li (2013) also demonstrate a PAC-MDP algorithm with improved sample efficiency bounds through transfer across similar tasks.”
• Multitasking Inhibits Semantic Drift (2021, methodology): “Following D’Eramo et al. (2019), we provide both training strategies with a fixed budget of training experience across environments : both RL joint and RL multi have been trained on the same number of game episodes per training environment.”
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control (2020, methodology): “Instead of designing a separate DNN triple for each task like those in related works [1, 2, 15, 21], we choose to pad the state and action with appended zeros such that they have fixed lengths.”
• Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “…and Borbely, 2008; Zhang, 2015; Lounici et al., 2009), but aside from a small pool of recent work (Shui et al., 2019; Ndirango and Lee, 2019; D’Eramo et al., 2020; Wu et al., 2020; Zhang et al., 2020; Bettgenhäuser et al., 2020), there is a lack of theoretical understanding of MTL with deep…”