Knowledge Transfer for Deep Reinforcement Learning with Hierarchical Experience Replay

By Haiyan Yin, Sinno Jialin Pan. 2017

124 citations recorded by the source corpus.

The process for transferring knowledge of multiple reinforcement learning policies into a single multi-task policy via distillation technique is known as policy distillation. When policy distillation is under a deep reinforcement learning setting, due to the giant parameter size and the huge state space for each task domain, it requires extensive computational efforts to train the multi-task policy network. In this paper, we propose a new policy distillation architecture for deep reinforcement learning, where we assume that each task uses its task-specific high-level convolutional features as the inputs to the multi-task policy network. Furthermore, we propose a new sampling framework termed hierarchical prioritized experience replay to selectively choose experiences from the replay memories of each task domain to perform learning on the network. With the above two attempts, we aim to accelerate the learning of the multi-task policy network while guaranteeing a good performance. We use Atari 2600 games as testing environment to demonstrate the efficiency and effectiveness of our proposed solution for policy distillation

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computational Theory and Mathematics

Citations in the corpus

Selective Experience Replay for Lifelong Learning (2018, background): “Some approaches include training separate networks for each task (Rusu et al. 2016; Yin and Pan 2017) and discovering action hierarchies (Tessler et al.”
Block Contextual MDPs for Continual Learning (2021, background): “Previous works have looked at multi-task and transfer learning from the perspective of policy transfer (Rusu et al., 2015; Yin & Pan, 2017), policy reuse (Fernández & Veloso, 2006; Barreto et al., 2016), representation transfer (Rusu et al., 2016; Devin et al., 2017; Andreas et al., 2017; Sodhani…”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “Atari, Mujoco [76] LFD POfD Atari [71] LFD Go [68] LFD AlphaGo Keepaway [50] Mapping RoboCup [97] Mapping Atari [86] Policy Transfer Distillation Atari [87] Policy Transfer Atari [89] Policy Transfer 3D Maze [91] Policy Transfer Dota2 [144] Reward Shaping”
Distilling Policy Distillation (2019, unknown): “Many RL distillation frameworks set up knowledge transfer as a supervised learning problem [23, 24, 9, 33], by following updates in the direction of: Eπ [∑|τ | t=1∇θH (π(τt)‖πθ(τt)) ] , with Monte Carlo es-”
Streaming Graph Neural Networks via Continual Learning (2020, background): “Besides, some studies [21, 32] have pointed out that the importance of different samples is different.”
Dual Policy Distillation (2020, unknown): “Policy distillation has achieved great success and led to stronger performance in challenging domains [ Teh et al. , 2017; Yin and Pan, 2017 ] .”
Transferring Autonomous Driving Knowledge on Simulated and Real Intersections (2017, background): “Recent work in deep reinforcement learning has looked at combining networks from different tasks to share information (Rusu et al., 2016; Yin & Pan, 2017).”
Attentive Multi-Task Deep Reinforcement Learning (2019, background): “[30] improve policy distillation by making the convolutional layers task specific and by using hierarchical experience replay.”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, background): “Yin and Pan [85] likewise introduced another policy distillation architecture to apply knowledge transfer for the deep RL.”
Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory (2017, background): “Generally, other scenarios might instead beneﬁt from sharing other parts of the network (e.g., Yin & Pan (2017)).”